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ABSTRACT 


Throughout history, military warriors have used games to accomplish 
training objectives. Recently, personal computer-based games have emerged as 
viable training platforms. In this research project, we evaluated the training 
effectiveness of simulation games using a particular proprietary first-person view 
tactical trainer called Virtual Battlespace 2™. Specifically, we examined squad 
level tactical maneuver of a combat convoy in a semi-permissive environment. 
We found that personal computer-based gaming was at least as effective as 
traditional training methods such as the sand table for preparatory tactical 
training. We found that trainees felt better trained after operating in the virtual 
environment. We also conducted an experiment to determine the extent to which 
the training of the simulation controller influences the effectiveness of the 
simulation. We found that the facilitator role can detract from a trainer’s focus 
and that the trainer’s practice and experience greatly affect the simulation 
training. Our findings justify the use of personal computer-based games for small 
unit tactical training. We conclude that personal computer-based gaming at the 
unit level can be a training multiplier, but the capability of the unit trainers to 
administer virtual training plays a large role in determining the effectiveness of 
the training tool. 
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I. INTRODUCTION 


A. MOTIVATION 

In 2006 and 2007, I led the logistics section of a training battalion at the 
Marine Corps’ western amphibious training base at Camp Pendleton, CA. I had 
just returned from combat duty in Iraq. I was in good company, because virtually 
every Marine in the battalion had just completed one or more combat tours. The 
Marine Corps used the training battalion as a resting place for deployment-weary 
Marines. From the Mayberry-like confines of our cantonment area, we were 
swept off the stage of constant deploying, training, fighting, and standing guard. 
Suddenly, we were watching the war the way most everyone else does. We read 
about the units whose ranks we once filled, and the Marines with whom we once 
served, as remote headlines in the newspaper or on the evening news. We no 
longer felt that insight of understanding the war from a first-person perspective 
because we were there seeing it and feeling it. Rather, we joined the millions of 
other people reading or hearing snippets of military action, distilled through a 
long journalistic chain. One did not have to read deeply or listen to too many 
evening news reports to gather that the face of the conflicts was ever changing, 
casting our experience quickly and surely into irrelevance. Within months, our 
experiences were merely anecdotal tidbits from a time that had come and gone; 
war stories to be told one day, like those of World War II or Vietnam. We had 
been there and done that, but we had become spectators. Worse yet, our skills 
declined. We were training recruits, administering the daily dither of a home-site 
organization. We were not fighting anyone or thinking about fighting anyone, 
other than through the abstractions of firing-range targets. We could feel our 
warfighting skills erode just as our experiences melted into memories. We were 
becoming irrelevant. 

I had been in the command about six months when my assistant, a well- 
educated Captain who had joined the Marine Corps after 9-11 to serve his 
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country, approached me with a suggestion. He couched his discussion with an 
overview of our waning warfighting skills and a description of the hodge-podge of 
logistics Marines in our section. He suggested that we go to the convoy trainer. 
At the time, my knowledge of the convoy trainer was limited to some vague 
understanding of simulation, and I had no idea where it was or who operated it. I 
liked the Captain’s idea, though, so he and I started working on a plan when we 
had time, gradually forming it into some sort of executable concept. 

First, we found the convoy simulator. I Marine Expeditionary Force (MEF), 
for all intents and purposes, encompasses all of the active duty Marines west of 
the Mississippi, and it is headquartered near the beach on the southern end of 
the base. Tucked right behind the headquarters building is the I MEF Simulation 
Center, a modern box of a building partitioned inside to cubicle cells of computer 
activity. We met with the staff there, and they showed us the convoy trainer, 
which turned out to be a bunch of laptop computers spread through a few 
different spaces. I had been in the Marine Corps long enough to know that 
anything new came with its own acronym soup, and soon I was lost in VBS, 
DVTE, and a variety of other terms. Eventually, I figured out that the convoy 
trainer was hosted on Virtual Battlespace 2 {VBS 2™), a networked personal 
computer-based first-person shooter game with civilian roots. The Simulation 
Center staff could use simple drag-and-drop controls to build any scenario I 
wanted on any one of three different selections of terrain. Quickly, I saw that we 
had a very versatile environment with a setup that would allow all of my Marines 
to participate as individuals. 

Through the next six months, I worked with my assistant to develop and 
administer a series of classes on basic convoy concepts and Tactics, 
Techniques, and Procedures (TTPs). We culminated the series with a detailed 
higher headquarters operational order to the group to conduct a convoy. We had 
built the exercise scenario on paper and provided the details to the Simulation 
Center staff for programming. We had our Marines develop different plans for 
conducting the mission and selected a few of the best for practice in the 
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simulator. After all the preparation work was complete in the fall of 2007, the 
whole section went to the Simulation Center. The staff there gave the Marines 
about an hour of instruction. The first half was administrative in nature with 
standard Marine rules for the obvious such as keeping drinks away from the 
computers. The second half, about 30 minutes in length, taught the Marines how 
to be themselves virtually. Drivers learned to move, turn, and look with keyboard 
keys and gunners learned to shoot with a mouse. Finally, we had a team of 
virtual warfighters, complete with a plan. The Marines executed the mission 
once before lunch and once after with comprehensive After Action Reviews 
(AARs) after each run. 

I left the Simulation Center with questions burning in my head. As a tank 
officer, I had used simulation all of my career, particularly for tank gunnery. I had 
used the Army’s Close Combat Tactical Trainer (CCTT), where huge tank mock- 
ups complete with all four crew positions are networked together to fight on a 
virtual battlefield. However, those simulations looked and sounded like tanks, 
albeit with many deviations from the real thing. Our convoy exercise at the 
Simulation Center was different. The Marines did not see anything that day that 
looked remotely like a High Mobility Multi-Wheeled Vehicle (HMMWV), a 
machine gun, or a rifle. They saw laptop computers, and that was it. I had seen 
personal computers used for simulation before, but it was always through the 
very abstract level of operational or strategic warfighting, such as TACOPS, 
where map icons represent all actions. In the Simulation Center that fall day, we 
were not having blue rectangles meet red diamonds and wait for math to crunch 
through the computer to see what was left. The Marines were fighting as their 
individual selves, quite realistically represented in the virtual world. They had 
gone through two relatively elaborate exercises in less than six hours with no 
logistics footprint, no equipment, and no permanently established bulky 
simulator. 

I mulled the experience over for the next few weeks, but I considered it 
much more thoroughly after a discussion with one of the Marine Corps’ few 
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Simulation Officers. He explained that the Marine Corps would field VBS to all 
infantry battalions within a couple years. Soon, the VBS exercise I had executed 
with my Marines could be the norm for one-third of the Marine Corps. I looked 
back on my experience with alarm, recognizing that I had known very little about 
the simulation throughout the whole ordeal. I was not even certain the Marines 
had gained that much from the simulation. They benefitted from the classes a bit 
and certainly the planning forced them to get into a combat mindset for a moment 
in time. Overall, the program was better than nothing; at least they had been 
forced to put their minds back in a combat scenario. Had the VBS simulation 
contributed to making the Marines tactically better at their jobs? Or, worse, had 
the simulation taught them things that were not right in the real world? Certainly, 
we had experienced “game-isms,” peculiarities associated with the simulation 
exercise. Did VBS really make them better? I was skeptical. 

Once I reported to the Naval Postgraduate School, I started to discuss this 
idea with faculty and was surprised at how little knowledge existed about VBS. 
Most people only knew about the simulation from a cursory back-of-the-brochure 
sort of viewpoint, and no one had really looked into how effective the simulation 
was or what skill sets it best suited. The Marine Corps had already spent a lot of 
money on the system and had scheduled its distribution to the fleet. I became 
fascinated by what was known, and equally by what was not known, about the 
system. Finally, my thoughts and experience with VBS boiled down to a single, 
simple, burning question: does VBS make Marines better at their jobs? I 
decided to devote my two years of modeling and simulation study to this 
question. 

B. SCOPE 

I originally administered a platoon-level convoy exercise because it fit the 
needs of my Marines, but the proliferation of lEDs throughout OEF and OIF has 
brought this scenario into the public and military consciousness as never before. 
Figuring out how to tactically drive around has become one of the cornerstone 
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issues of Post-9-11 military tactical thought. Moreover, this exercise focuses on 
the senior NCO or junior Staff NCO on whom so many important war decisions 
rest. Small unit tactical convoy is certainly an exercise that trains Marines in a 
skill set with both operational and tactical relevance. 

We chose the platoon-level convoy because of its deep relevance in 
current operations. Convoy operations, once considered a rear echelon low- 
intensity combat function, have come to the forefront of military thinking in OEF 
and OIF. Many of the war’s casualties have occurred in this area. Tactical 
convoys are often planned and executed at the platoon level, and platoon 
leadership can be required to make life and death decisions in tough situations. 
The operational forces have quickly ramped up convoy training since the 
beginning of OEF, and this aspect of simulation training deserves immediate 
critical scrutiny to provide the best answers for the Marines and soldiers on the 
ground. 

The primary purpose of this thesis is to explore the effectiveness of VBS 
2™ as a training tool for small unit tactics. Additionally, the thesis seeks to 
investigate whether the simulation itself contributes to increased training or if the 
trainer, influenced by training and experience, affects the training impact of the 
simulation. Information gained from exploration of the second question is 
intended to suggest best implementation practices for the simulation, assuming 
that it is shown to be effective. Flowever, the thesis does not intend to generalize 
findings broadly, but rather focuses on platoon-level tactical convoy training. 

C. RESEARCH QUESTIONS 

This thesis focuses on the following questions: 

• Does preliminary training with VBS 2™ positively impact 
performance in a squad level tactical scenario? 

• Does the level of training of the simulation controller impact the 
effectiveness of VBS 2™ preliminary training? 
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D. METHODOLOGY 

1. Background Study 

We started the project by researching how other people had tackled the 
problem of determining whether personal computer games work for training. We 
conducted a literature review focusing on two major topic areas: the historical 
use of games for training and training transfer of simulation based training. This 
background research served as a starting point for the project’s work. 

2. Pilot Studies 

We started this project with no technical knowledge of VBS 2™. We set 
up some preliminary studies to learn how to work the software and develop a 
basis of experience for intelligent use of the simulation as a training tool. We 
started by learning how to operate the simulation as a user and how to teach 
others to do the same. We then learned how to develop scenarios in VBS 2™ 
and tested whether they were working as designed. To see whether they 
worked, we first tested whether the scenarios technically performed as intended. 
Then we conducted testing to determine whether the scenarios met their tactical 
objectives. 

3. Experiment in Support of Simulation Effectiveness Question 

Once we had developed a sound base of experience to use VBS 2™ for 
training, we developed a set of scenarios to support training. We administered 
the training to an operational unit and used several methods to determine the 
effectiveness of the training. Methods of evaluation included surveys, 
performance on academic knowledge tests, and performance in live training. 
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4. Experiment in Support of Trainer Effect on Effectiveness 
Question 

We developed a training program to train the trainer on the use of VBS 
2™ to accomplish unit goals. We administered the training program and used 
survey and academic knowledge test evaluations to determine whether the 
trainer made a difference to the effectiveness of the simulation training. 

5. Conclusion 

We concluded the study with a series of observations and 
recommendations. We also described some future research that could be 
developed from this project. 

E. CHAPTER OUTLINE 

I. Introduction 

A. Motivation 

B. Scope 

C. Research Questions 

D. Organization of the Thesis 

II. Background 

A. Introduction 

B. Evolution of Personal Computer-based Training Games 

C. Training Transfer 

III. Preparatory Experiments 

A. Interface Familiarization Pilot Study 

B. Scenario Development Pilot Study 

IV. Experiment in Support of Hypothesis 1 

A. Introduction 

B. Method 
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C. Results 


D. Discussion 

V. Experiment in Support of Hypothesis 2 

A. Introduction 

B. Method 

C. Results 

D. Discussion 

VI. Recommendations and Conclusions 

A. Recommendations 

B. Future Research 
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II. BACKGROUND 


A. INTRODUCTION 

The U.S. Marine Corps encourages extracurricular reading through the 
“Marine Corps Professional Reading Program.” This list of military related books 
divides its recommended titles by rank so that each Marine reads material 
correctly related to the level of warfare typical of his or her pay grade. For the 
most part, these titles take Marines back into the history of warfare or military 
heritage. They may provide doctrinal insights or relate different views of tactics, 
techniques, and procedures. One book recommended for entry-level Marines at 
both the officer and enlisted levels stands out because it simply does not fit with 
these other works. The book is a science fiction novel from the mid-1980s called 
Ender’s Game (U.S. Marine Corps, n.d.). 

Orson Scott Card published Ender’s Game in 1985, when computers were 
not quite established in every household and the Cold War still defined the 
American understanding of what war was and what a military did. The novel’s 
protagonist, Ender, was a 12-year-old boy who played video games well. He did 
so well, in fact, that the government sent him to Battle School for advanced 
training. While at Battle School, he learned all the known techniques of playing 
the school’s games and became the best at them all. The government 
developed Battle School to immerse students in the gaming environment in 
preparation for armed conflict with the buggers, an insect-like alien race that 
threatened to overwhelm Earth. Ender excelled in this environment, taking on 
increasingly more difficult simulations. Finally, Ender fought in a culminating, 
enormously challenging computer game. Feeling his options close in after a 
particularly overwhelming series of actions, Ender made a risky move in the 
game and consequently won the day. Exhausted, he left the simulator to find out 
the secret of the game and of Battle School itself. His mentor and instructor 
explained: 
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Ender, for the past few months you have been the battle 
commander of our fleets. This was the Third Invasion. There were 
no games, the battles were real, and the only enemy you fought 
was the buggers, (p. 296) 

Ender, in fact, had been remotely controlling a fleet of ships fighting the real 
buggers. Some of his old friends from Battle School had been piloting the ships 
under his command (Card, 1985). 

This thesis is about simulation, so the idea of including a book in the 
Professional Reading Program whose theme espouses such a triumph of 
simulation is most intriguing. However, Ender’s Game offers more as a reading 
list recommendation than this. The Marine Corps most likely included this book 
as an illustration of the close, and sometimes indistinguishable, relationship 
between training and war. The idea of Ender unwittingly controlling a living, 
breathing, and dying army while playing a video game is novel and thought- 
provoking. He was training for war. Ultimately, every commander yearns for this 
level of training in any given scenario. That is, the training should be so real that 
one cannot tell the difference from actual combat. The military should seek this 
level of training in all endeavors, so that servicemembers execute in exactly the 
same way during peace or war with the state of hostilities being irrelevant. 

Michael Macedonia, the technology officer for Program Executive Office 
Simulation, Training, and Instrumentation Command (PEO STRI), looked at the 
inseparability between training and the reality of war another way. He hailed a 
scene from the movie Patton as illustrative of his point. In the scene. General 
Patton approached an open area while directing his driver to a battle scene. He 
described to his driver a scene of ancient battle between the Carthaginians and 
the Romans far removed from the movie’s World War II engagements. Then, 
Patton said, “I was here.” Macedonia pointed out that the scene painted a 
picture of a man so engrossed in the study of war history that events of his 
studies became their own reality in the general’s mind. General Patton’s literary 
education created memories in his mind that were not there before. Macedonia 
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explained that simulation boils down to the ultimate objective of creating 
memories. Soldiers and Marines on the battlefield against a fighting enemy 
should look at their situation and recognize that they were there, that they have 
seen this before (Halter, 2006). In this way, Macedonia sets the bar for 
simulation, games, and training. An individual who has used the training tool 
should approach the real event knowing its subtleties and nuances from thorough 
exploration with the tool. Like Ender, the soldier or Marine should participate at 
such a level that the fact of a real adversary becomes immaterial; the skills and 
their application are the same. 

This thesis invites exploration into two general domains. First, military 
games have a rich history and that history must hold some nuggets of truth about 
what games can and cannot do for the warfighter. Second, concern over the 
effectiveness of simulation is not a new topic. Studies wrestling with such 
questions are typically lumped under the umbrella of “training transfer.” A review 
of past training transfer efforts should shed some light on the Best Practices for 
evaluating simulation training systems. 

B. EVOLUTION OF PERSONAL COMPUTER-BASED TRAINING GAMES 

Humans have a history of playing games of the mind and of the body that 
seems as old and as diverse as any other activity. Today, an array of bright, 
shiny, new objects stands before the services as the cutting edge, innovative 
training of the future. Personal computer-based simulations are these objects, 
and they have gained attention, both skeptical and enthusiastic. Some 
researchers have called this genre of training device “serious games,” to thwart 
the idea that our troops are going to the arcade to learn to fight. One might 
question just how new and innovative this idea really is. 

1. Games Have Been around a Long Time-Just Like War 

The story has it that the father of a young Indian prince died before he 
could teach his son the skills, tactics, and strategies of war. The prince’s 
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advisors fretted over how to prepare the prince for battle with known adversaries 
that imminently threatened the kingdom. At a loss for a better training 
mechanism, the advisors introduced the young prince to chess, a game that he 
played regularly. Later, after a victorious rout on the field of battle, the prince 
attributed his success to the development of his mind with the chessboard 
(Forbes, 1860). The story is a mere anecdote, certainly of questionable value as 
historic fact. However, the idea that someone thought up the story illustrates the 
long-standing close link between games and war. The origins of chess provide 
grounds for debate, but the game clearly has roots in an Indian game called 
Chaturanga. In that game, the pieces were modeled after the battlefield assets 
of the day, such as elephant and horse, and two players took turns moving about 
a gridded square according to movement rules similar to chess (Caffrey, 2000). 
One simply cannot ignore these martial roots. 

A strategy game called Wei Chi, or, in popular western parlance. Go, is 
hailed as the oldest strategy game still played in its original state. Archaeological 
evidence points to ancient Chinese origins. The game of Go spread across the 
entire Orient and is still played. Legend suggests that the game has military 
origins. Encirclement is a fundamental feature of the game, and the ancient 
Chinese used this strategy of dominance both in hunting and warring tactics. Go 
is notable for many reasons. The game was extremely widespread and enjoyed 
a long history that still lives on. Go has been recognized as a domain of the elite, 
and proficiency in Go has been a criterion for the elite in China, Japan, and other 
Asian countries. In fact, Japan formalized Go as a fundamental part of the 
Shogun’s court in the medieval period. Most importantly. Go influenced many 
aspects of life in Asia including religion, philosophy, politics, and war. Go is often 
linked with the yin and yang of Taoism. Go influenced such thinkers as Sun Tzu, 
the author The Art of War, the oldest military doctrine, which is a book that 
serves players of Go as well as warfighters. Mao Tse-tung reputedly used Go in 
the development of his military and political theories. Go’s importance stems 
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from its widespread impact on a whole society throughout history, not only as a 
pastime, but as a way of thinking about life and applying those ideas practically 
(Shotwell, 2003). 

These games from ancient history tell us very little about how to use a 
game to train a military fighting man today. The depth of the history, however, 
indicates that games have been used to develop and influence military strategy 
far back into the beginnings of war itself. The idea of abstracting the concepts of 
war into a game, with systems of rules and randomization that individuals can 
play out against other minds, is not new in any way. Techniques have matured 
over the years, as the next sections will show, but the theme of gaming for 
military benefit has deep roots. One can trace the games just discussed back to 
the fifth century BC. The serious game is a new term, but it is not a new idea. 

2. Games Do Not Have to Be Limited to Strategic Levei 
Abstractions 

Gaming for training began to evolve beyond the confines of the 
chessboard in the mid-1600s. In England, the King’s Game was used as a 
training device. With thirty pieces to a side and a more robust rule set than 
chess, the game added realism beyond chess. Meanwhile, the French used card 
games to train military professionals. These games featured actual engagement 
situations, thereby extending realism beyond the chessboard. The Prussians 
also used games for training, relying on the math and geometry of the game to 
replicate war as a science (Brewer & Shubik, 1979). In fact, it was a Prussian 
named Baron von Reisswitz who is credited with fathering modern wargaming. 

The Germans coined the term kriegsspiel, translated “wargame” in 
English. No thesis written by a Marine would be complete without a tribute to a 
dead German, and this one will be no different. Baron von Reisswitz served the 
state of Prussia as the war counselor at Breslau. In 1811, he introduced a 
training concept that has changed very little over time, considering the 
technological breakthroughs of the two centuries since then. Officials moderated 
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a game played on a table with actual terrain modeled on top. Units represented 
by blocks moved across the table in response to orders from the players. The 
moderating officials received the orders from the players and updated the table 
accordingly. In order to adjudicate battlefield effects, they used tables to 
calculate attrition based on factors such as range and terrain. The game was not 
deterministic though, because the final outcome faced the uncertainty of a dice 
throw. Many of the advanced and highly used simulations of today owe their 
conceptual design to this early innovation, with the only significant difference 
being the added computational power of modern computers. Nevertheless, for 
all of the evidence of Reisswitz’s contributions in today’s war simulations, 
Reisswitz did not revolutionize warfare, battlefield planning, or combat training in 
1811. Kriegsspiel merely served as an amusement for the elite, never to be used 
by the fighting forces on the ground. Baron von Reisswitz’s son brought 
kriegsspiel to the common fighting man when he altered the game so that players 
could use a map, making the game much more portable. In 1824, the Prussian 
chief of staff General Karl von Muffling liked the idea so much that he ordered all 
garrisons to use it (Caffrey, 2000). 

In 1837, General von Moltke took the ideas of Baron von Reisswitz and 
his son and turned them into a bona fide training regime. He incorporated 
wargaming with staff rides to train the officer corps of the Prussian army. As the 
Prussians demonstrated prowess on the early nineteenth century battlefield, the 
other nations of the world took notice and started using wargames of their own. 
Spenser Wilkinson, a British college student vacationing in Germany in 1873, 
exemplified this world attention. After noticing Britain’s numerical inferiority to the 
rest of Europe in a pamphlet, he returned to England and organized the country’s 
first wargaming club. In the United States, Major Livermore started copying 
German wargames in 1883. He improved the Germans’ attrition tables with 
updated statistics from the American Civil War. The U.S. Army’s Chief of Staff, 
General William T. Sherman, disapproved of the wargame because, in the game, 
units fought to the last man, something experience had proven unrealistic. About 
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the same time, William McCarty Little fathered naval wargaming in America. A 
key player in the formation of the Naval War College, McCarty Little instituted 
wargaming there in 1889, where wargames have occurred every year since. The 
Army followed this practice in its own war college starting in 1899 (Caffrey, 
2000 ). 

Wargaming demonstrated both its power and its shortcomings in the 
preparations for World War I. Both the Germans and the British analyzed Count 
Alfred von Schlieffen’s invasion plans, and both governments used wargame 
results to make national decisions about military preparation. However, the 
German wargaming, meticulous and calculated as it was, failed to anticipate 
diplomatic and political consequences that were seen in their defeat. While most 
of the nations who participated in World War II used wargaming in the inter-war 
years, the Navy and Marine Corps had the champion effort. The Navy developed 
the island-hopping theory that would eventually become the backbone of the 
Pacific campaign, while the Marine Corps changed the face of amphibious 
doctrine through wargames in lieu of a budget for live amphibious training. By 
far, the most significant wargamed event of this time was the Japanese attack on 
Pearl Harbor, where wargaming radically changed the original plan, producing 
the intended results. All major players in World War II used wargaming to some 
extent, such that wargaming was an accepted military practice by the end of the 
conflict (Caffrey, 2000). 

Reisswitz brought us the red and blue of war. He color-coded his original 
nineteenth century tokens this way, setting a standard for military lingo today. 
More than just colors, Reisswitz introduced a way of visualizing the battlefield 
and its infinite layers of possibilities. Unlike the strategic games of Go, chess, 
and patteia that were played at the strategic level in the homes of the elite, 
Reisswitz’s style of wargaming influenced the warfighter himself at the 
operational level of war. As military actions such as Pearl Harbor illustrated. 
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wargaming had much more power than providing a theoretical forum for abstract 
concepts: wargaming started to change war itself, as militaries instituted it as a 
standard battle preparation activity. 

3. Wargaming Links the Living Room and the Command Post 

“A gun capable of hitting a toy soldier nine times out of ten at a distance of 
nine yards” sounds like the perfect Christmas gift for a young boy. This quote 
comes from the exuberant text of H. G. Wells’ 1913 wargaming manual Little 
wars. Wells was not excited about this toy gun for his children; rather, he was 
using the toy along with paper houses, miniature trees, and toy soldiers to build 
his own battlefields on his lawn or living room floor. The roots of Wells’ work 
started with “lunching with a friend” and evolved into a sophisticated explanation 
of how to model war in one’s home, complete with a full example battle (Wells, 
1970). Robert Louis Stevenson is another notable author who engaged in war 
games, devoting much of a three-year convalescent period to the development of 
a detailed war. Sir Winston Churchill’s childhood war games with his brother. 
Jack, influenced his decision to join the military. Churchill never lost his love of 
playing with toy soldiers, taking the hobby into his adult years (Featherstone, 
1962). 

Toy soldiers emerged in European culture on the heels of Reisswitz’s 
wargame. Daddy played with lead figurines to re-enact history or to learn the 
best tactics, and Junior wanted a piece of the action. War made great toys, and 
World War I sparked its own set of miniature replicas of machines of destruction. 
In fact, toy soldiers grew into an industry to satisfy both youth and adults. This 
play has never gone away. From plastic “army men” to the Gl Joe action figures, 
youth have delighted in toys that depict war. FI. G. Wells captures it all in his 
subtitle to Little wars: “A game for boys from twelve years of age to one hundred 
and fifty and for that more intelligent sort of girl who likes boys’ games and 
books.” Many people want to be the general, if only for a moment, whether it be 
the boy in his room, the adult hobbyist at his couch, or the actual general himself 
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making plans to defeat the enemy (Featherstone, 1962). This fundamental 
inseparability between the living room and the war room contributes to 
understanding the strengths and weaknesses of the twenty-first century serious 
games. 

4. A Soldier Can Interface with an Electronic Device to Train 

In the 1930s, Edwin Link did not have money for a plane to learn to fly, so 
he developed a plan to help people learn to fly without an actual aircraft. He built 
a simulator. His wooden box mounted on a crude motion platform did not look 
much like its modern successors of today, but it provided enough realism for 
people to learn the basics of aviation, particularly in the area of instrument flight. 
Link’s work might have gone largely unnoticed if an unfortunate series of Army 
Air Corps accidents had not attracted attention to flight training. In 1934, Link 
amazed Army officers by safely landing his plane out of a cloudy sky by flying 
completely on instruments. The Army Air Corps contracted with Link to build 
simulators for the military so that pilots could learn instrument flying. Link’s 
trainer contained a mock instrument panel and the basic plane controls. With 
these tools, students could learn to fly without looking at the ground. Link’s 
company produced over 10,000 of his “Blue Boxes” during World War II, and 
simulation became a staple of military pilot training {L3 Communications, 2010). 

Link’s trainer has very little to do with gaming itself, but it is relevant to this 
study because of its introduction of the virtual world. Until the twentieth century, 
training at the soldier level involved learning basic skill sets, such as operation of 
a firearm or short scripted movements in the form of battle drills. World War I 
changed all that with complicated machinery, like the tank, machine gun, and 
airplane; equipment that could influence the battlefield like never before. 
Suddenly, an individual soldier’s relationship to a particular machine mattered. 
The idea that the military needed to train this relationship at all was novel, and 
Link’s idea of doing so through simulation was truly ahead of his time. Though 
crude. Link’s trainer, through its instruments and motions, provided the trainee a 
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visualization of a world that did not exist. Link invented the virtual world, before 
the computer made such things commonplace. It is worth noting that the first 
virtual world was developed for military training. 

5. The Computer Adds New Possibilities for Wargaming 

After World War II, concern over how to fight a nuclear war gave 
wargaming a new boost, since this was the only venue available to analyze 
nuclear tactics and strategy. During the 1950s, the Air Force led the Department 
of Defense to work with RAND Corporation to develop wargaming as a complex 
mathematical exercise. RAND created some ideas that stuck, such as the 
gridded playing board, tables to calculate the results of combat actions 
mathematically, randomization through dice, rules for playing by turns, and the 
incorporation of terrain. Charles Roberts conducted a parallel game 
development effort that led to the creation of the commercial entertainment 
company Avalon Hill, in 1958. RAND brought mathematical gaming to the 
military, and Avalon Hill took it to the living room (Smith, 2008). Through the 
1950s and 1960s, this mathematically based style of military wargaming 
matured. Most notably, though, wargaming remained mainly at the strategic or 
operational levels, with games oriented on winning by destroying more personnel 
and materiel than the other players destroyed. There was some recognition that 
wargaming needed to extend beyond attrition. For example, a 1964 Advanced 
Research Projects Agency (ARPA) project sought a wargame that would 
demonstrate political, economic, and psychological aspects of an insurgency, but 
the research efforts did not go very far in changing the face of attrition gaming 
(Caffrey, 2000). Such wargaming efforts continued to mature throughout the 
1970s, in both military and civilian sectors. While military and civilian gaming 
remained largely separate efforts, it would be a mistake to discount any 
relationship between the two. For example, in 1974, the Army purchased a 
commercial wargame called Fire Fight to supplement its wargaming collection 
(Caffrey, 2000). 
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These mathematical models of war increased in accuracy as the math 
increased in complexity. Eventually, computers broke the mold for wargaming. 
At first, computers served only as glorified calculators, but even this advance 
allowed wargaming complexity to develop. The mechanics of game play 
remained the same with playing boards, lookup tables, representational pieces, 
and dice, but computers sped up the calculations allowing more precise 
numerical modeling. During the 1970s and 1980s, the wargame moved from the 
playing board to the computer itself as computers grew powerful enough for the 
job. Eventually, computers supported gaming well enough to offer advantages 
never fathomed in the days of the playing board. Players could work in separate 
rooms with customized views. They could make decisions and implement them 
in real time with many others doing the same. The age of the computer game 
had begun. Because computers were still too expensive and bulky for the 
average consumer, these games stayed military. The games’ roots in strategic 
and operational board gaming kept simulation at the higher levels of war. 

Wargaming and war modeling became integral components of training 
and planning during the 1990s. The rise of the personal computer made 
simulations such as JANUS and Modular Semi-Automated Forces (ModSAF) 
widespread. Simulations like these proved themselves during the planning and 
execution of the Gulf War (Smith, 2008). All services used such games in 
training throughout the 1990s, and many joint efforts brought the services 
together in simulated environments. The 1991 creation of the Defense Modeling 
and Simulation Office (DMSO) demonstrated the importance of the community, 
and simulations such as the Joint Simulation System (JSIMS) and the Joint 
Warfare System (JWARS) showed efforts to bring everyone together in the joint 
arena (Caffrey, 2000). 
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6. 


The Military Can Train Cheaply through Adaptation of Civilian 
Games 


In 1995, General Krulak, the U.S. Marine Corps Commandant at the time, 
issued his Planning Guidance that provided the broad vision for the service for 
the next four years. The following directive ranked third-highest in the priority list: 
“Make our education and training processes and institutions technologically 
innovative, challenging, and fun” (Jernigan, 1997). General Krulak even added 
teeth to this task with the creation of the Marine Warfighting Lab to test new 
strategies, technologies, and ideas. The Commandant’s Planning Guidance 
opened new horizons for Quantico’s Marine Corps Modeling and Simulation 
Office (MCMSO). A couple of Marines searched all available civilian gaming 
systems to find anything useful for military training. As a marketing ploy, id 
Software had released parts of Doom II as shareware for players to modify. 
Seizing the opportunity. Sergeant Daniel Snyder spent three months of his off- 
duty time changing Doom II into Marine Doom, an application with military 
avatars. Ml 6 rifles, and realistic survival rates (Jernigan, 1997). As many as four 
players could fight each other in a two-dimensional urban environment of walls 
and passageways. For the price of a $49.95 Doom II CD-ROM, the Marine 
Corps had built a simulation capable of teaching individual and fire team skills. 
Since the game resided on a CD-ROM, Marines could use it on any computer, 
playing the game at their desks or taking it on deployment wherever the Corps 
might send them (Riddell, 1997). 

Marine Doom did not come into the Corps as just a fly-by-night affair. 
While the program owed its existence to the innovation of fewer than a handful of 
Marines, the project reflected imagination, analysis, and forward thinking. Merely 
gaining acceptance of any such game was a Sisyphean task. Ironically, 
Sergeant Snyder had installed Doom on the hard drive of his computer a few 
years earlier in communications school. In doing so, he had violated a Marine 
Corps order prohibiting computer games on military hard drives and received 
nonjudicial punishment for his actions. In 1995, General Krulak’s guidance 
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changed everything, and Marine Doom’s prospects looked bright with the full 
support of the Commandant. Sergeant Snyder and his Officer in Charge, 
Lieutenant Scott Barnett, had developed a training system that met Marine Corps 
training objectives of interlocking fields of fire, use of cover and concealment, and 
effective communications (Riddell, 1997). Marine Doom joined a small list of 
twelve simulations allowed on Marine computers (Jernigan, 1997). For the next 
few years. Marines fought each other in their offices during lunch breaks. The 
program slowly slid into extinction as better technologies developed. Game 
problems, such as player tunnel vision, unrepresented firing positions, and 
weapons that never overheated, came to the surface (Jernigan, 1997). Marine 
Doom only featured a two-dimensional battle space, a particularly limiting 
problem when simulating urban combat. Lurking in the background of this early 
experiment was the haunting question of how much good such technology could 
do and, worse, whether it could do harm by teaching players the wrong 
techniques. With such a rags-to-riches start, eventually gaining the enthusiastic 
support of the Commandant himself. Marine Doom soared to great heights as a 
training platform, but faded into obscurity almost as rapidly. In the end, 
commanders did not see the training results. Two Marines had successfully 
leveraged commercial off-the-shelf (COTS) technology, but the training simply 
did not transfer to the warfighter. The game disappeared. The ground had been 
broken, however; civilian computer gaming technology could be levied for military 
use. 

General Krulak’s guidance, coupled with Sergeant Snyder’s proactive 
execution, had demonstrated the potential for using the shelves of the local 
computer store for military training, but Sergeant Snyder was not the first to use 
this technique. In 1980, Atari introduced the first-person shooter game with 
Battiezone. This early video game allowed a player to act as a tank gunner and 
engage enemy tanks on a three-dimensional battlefield. The player looked 
through a small viewfinder window and manipulated controls like those in a real 
tank. Users could prowl around a virtual world, albeit one composed of the 
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sparse graphics of the time, complete with three-dimensional terrain. Battlezone 
simplified the battlefield into sterile tank on tank battles with no people, just tanks 
that either lived or died. Nevertheless, players immersed themselves in the 
soldier’s battlefield for the first time, fighting from the arcade just like a soldier in 
a real tank (Halter, 2006). 

The Army recognized the potential for the addictive game play to turn a 
leisure activity into valuable training. The service persuaded Ed Rotberg, Atari’s 
Battlezone designer, to transform the game into a military training device. After 
three months of long hours and hard work, Rotberg produced Army Battlezone. 
General Donn A. Starry was the commanding general of TRADOC at the time, 
and he oversaw Atari’s project. General Starry recognized that soldiers had 
grown up in an environment of electronic gadgetry, and teaching methods for this 
new breed had to be developed accordingly. In a 1981 conference, he asked a 
question that is still quite relevant: 

In an era that has seen such fantastic technological achievements, 
how is it that our soldiers are still sitting in classrooms, still listening 
to lectures, still depending on books and other paper reading 
materials, when possibly new and better means for training have 
been available for many years? (Halter, 2006) 

General Starry had introduced the training concept that General Krulak worked 
so hard to put into practice fifteen years later. The products in the teenager’s 
arcade were fair game for military training, if only the military applied the right 
imagination and planning. 

7. Games Can Benefit the Individual Warfighter 

While the wargaming discussed thus far impacted strategy for centuries, 

this type of gaming does nothing for the individual warfighter. The soldier or 

Marine on the ground needs something at the tactical level. Before computers, 

the Tactical Decision Exercise (TDE) accomplished this objective. A TDE poses 

a static military situation to the participant in the form of a simple sketch or 

composite of map graphics. The participant individually develops a solution for 
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the situation, usually in a limited amount of time, and prepares the orders that he 
or she would give subordinates to execute his or her plan. IDEs orient on an 
individual’s decision-making ability. Rarely is the plan itself the point of the 
exercise; most often, the rationale that led to the plan produces the real learning. 
IDEs offer teaching power by pitting the mind of the participant against the 
experience of a mentor or instructor. Certainly, the TDE provides a forum to 
discuss tactics, techniques, and procedures in a practical application. More than 
that, IDEs provide a window into the mentor’s mind, allowing the student to see 
how to work through a problem or to see how the mentor thinks. In this way, 
IDEs provide a mechanism to convey commander’s intent and battlefield vision. 
Like the board games previously discussed, computers take IDEs to new levels. 
Virtual worlds allow participants to see their plans through a simulated execution. 
As Major Brewster puts it in his 2002 article on the subject, “Simulation allows the 
student to progress to the point where he can observe the ramifications of his 
decisions” (Brewster, 2002). An understanding of IDEs is crucial to harnessing 
the power of personal computer-based games. 

The Marine Corps uses a simulation called Close Combat Marine (CCM) 
to stimulate decision making at the junior leadership level. The simulation 
provides a simplistic two-dimensional map interface for multiple players to 
manipulate units, usually at the squad level. Using the map, a trainer can 
develop a scenario, and Marines can maneuver forces and watch the results of 
their decisions. Quite simply, CCM is a natural computer extension of the TDE. 
In 2007, two officers at the Naval Postgraduate School conducted a study to 
determine whether CCM introduced any training advantage over the more 
traditional TDE. They found that both training media provided benefit, but that 
only the computer simulation provided a means to evaluate situational 
awareness (Fitzpatrick III & Ayvaz, 2007). Because of the importance of 
situational awareness in team leadership and decision making, this finding 
indicated that computer games offered the individual soldier or Marine something 
that could not be obtained with just paper, pencil, and tokens on a table. 
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A simplified version of kriegsspiel has provided a training medium for the 
individual soldier or Marine for years. Sand table exercises involve the use of 
small replicas of battlefield forces on a model of the terrain to demonstrate, 
analyze, or practice tactical maneuver and fires. While this sort of gaming has a 
rich history and can become quite complex, implementations at the small unit 
level are usually quite simple. The terrain model might be a ruffled-up bed cover 
in a barracks room, a dirt patch in a common area outside, or an actual table top 
with sides holding a sand box that easily models hills and valleys. Force replicas 
might be simple toy military men and vehicles, miniature vehicle replicas, or 
rocks available on the ground. Typically, trainees practice a scenario by 
timeframe, with each key player describing actions he would take at prescribed 
times and locations. A controller can then assess the effects of the actions to 
determine how the succeeding timeframe will begin. In a study involving a 
comparison of the sand table exercise and other traditional training with 
computer-based games. Majors Nolan and Jones noted that some participants 
preferred the sand table to simulation because the sand tables “help lowest level 
operators get a big picture view” (Nolan & Jones, 2005). 

Small units may practice a tactical scenario by using a field or large 
common area in garrison for a physical walk-through of the exercise. The trainer 
uses markers on the ground to replicate the terrain from the scenario. Trainees 
then walk along the replicated routes to each point in the scenario, briefing their 
actions as they would around the sand table. Such exercises provide individuals 
the opportunity to visualize what is happening around them in the context of the 
scenario. Such exercises, therefore, provide a valuable opportunity to 
synchronize actions and provide all participants with a common view of the big 
picture. 

Returning the focus to computers, the gaming landscape changed for the 
individual warfighter in the past decade, but the military did not provoke it. 
Modern personal computers became powerful enough that the resulting civilian 
gaming market produced software that rivaled anything seen in the defense 
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arena. This brings the discussion back to 1995, to Sergeant Snyder and Marine 
Doom. While Sergeant Snyder’s project broke new ground, personal computer- 
based games did not become the training venue of choice in the late 1990s. 
Military culture still looked at gaming as a hobby or recreation rather than 
something to consider as a serious training aid. That culture began to erode as 
the profile of computer experience for the individual warfighter himself changed. 
By 2005, a reporter noted the prevalence of gaming in Iraq. Games such as Half 
Life 2 and Halo competed for storage space in the little deployment libraries 
reserved for books and movies (Associated Press, 2005). The handwriting was 
on the wall: gaming was a part of life for troops. The only issue was whether the 
military could glean training value from it. 

Michael Macedonia, a former Chief Technology Officer for PEO-STRI, has 
written about the change in military culture that has made computer gaming an 
acceptable concept. Using the catch phrase “wired generation,” Macedonia 
pointed out that the pool of young people currently filling the ranks of the military 
has always known computer games. He summarized some U.S. Army studies 
by highlighting the following differences in the wired generation’s skills and 
attitudes: 

• Multiprocessing, the ability to perform several tasks (such as listen 
to music, talk on a cell phone, and use the computer) concurrently 

• Attention span variation in a manner similar to senior executives 
exhibiting fast context switching 

• Information navigation changes that define literacy not only as text 
but also as images and multimedia 

• Shift in focus of learning from passive listening to discovery-based 
experiential and example-based learning 

• Shift in type of reasoning from deductive and abstract to the 
concrete 
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• Intelligence organized in easily accessible databases 

• Community of practice emerging from sharing tasks involving both 
tacit and explicit knowledge over a substantial period of time.” 
(Macedonia, 2005). 

Introduction of a loose term is necessary for the remaining discussion. 
Most people have a certain mental image when they hear the term “computer 
game.” Generally, most people imagine something played on a personal 
computer or similar device, such as XBox. This concept has broadened to 
include games that can be networked through wire connections between 
computers or across the Internet. At the extreme, massively multiplayer games 
stretch across the broad expanses of the Internet, encompassing thousands of 
users. For the most part, these games are leisure activities both in design and in 
use. The military uses games that fit into this general category for training or 
analysis. To distinguish computer gaming with an actual military purpose, Ben 
Sawyer, a high-tech freelance writer and technology consultant, coined the term 
“serious games,” and the term has caught on (Macedonia, 2005). For sake of 
convenience, this thesis incorporates the various sorts of personal computer- 
based military games under this umbrella. Serious games are based on personal 
computers, whether networked or not. They are games in which participants are 
actively engaged in a thinking contest against other participants or against the 
computer algorithms of the game. They are serious in that they are intended to 
have military value. For the purposes of this thesis, serious games have value 
for the direct user; in other words, the trainee himself sits at the computer and 
interacts with the game. 

Serious games can contribute to the military. Two Canadian researchers 
summarized potential uses for serious games in a paper prepared for the 2008 
Interservice / Industry Training, Simulation, and Education Conference: 

• Introducing, teaching, and rehearsing new drills; 

• Showing both enemy and friendly viewpoints; 
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• Representing the use and effects of future systems not yet 
available for conventional training; 

• Reviewing actions and events from different perspectives in post 
game analysis, after action review; and teambuilding (Roman & 
Brown, 2008). 

The authors further pointed out that serious games train cognitive skills 
but not psycho-motor skills. More importantly, the use of serious games is not 
just some theoretical proposal of the academics. Quite the contrary, serious 
games are already out there, and science is catching up to see what operators 
are doing with them. 

If serious games provide any benefit at all, their price tags attract 
immediate attention. Serious games are cheap, particularly if the civilian market 
bears a parallel research and development burden. The past decade has seen 
quite a bit of Department of Defense acquisition reform aimed at minimizing the 
massive, stove-piped, counterproductive Pentagon buying methods. Contractors 
with broad system goals design and produce, cutting costs where they see fit to 
produce the best product. Serious gaming provides the perfect arena for this 
type of acquisition. Civilian contractors often have experience in designing the 
type of software the military needs. The project simply becomes an adaptation of 
existing ideas to military requirements (Robinson, 1998). One of the first modern 
large-scale efforts toward adapting a civilian game to military needs was Mak 
Technologies’ modification of Spearhead in the late 1990s. Warren Katz, the 
chief executive officer, warned against oversimplifying the civilian game 
adaptation process: 

Many people think you can take a video game out of a box and just 
use it for training or think the modifications are small. The 
modifications are fairly sophisticated. Making a video game HLA- 
compliant is no easy task. (Erwin, 2000) 


Nevertheless, the cost of the development of serious games pales in comparison 
to large-scale simulations, and operational costs are virtually negligible compared 
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to those associated with large simulations or field training. Katz compared 
“expensive dome-based simulators with a motion base and full wrap-around 
imagery,” that cost $5,000 to $10,000 per hour to operate, to a personal 
computer-based game that costs 25 cents per hour (Erwin, 2000). Clearly, 
money talks when it comes to serious games. 

In 2004, with Operation Iraqi Freedom stealing the spotlight of military 
money and thought, the Defense Advanced Research Projects Agency (DARPA) 
fielded DARWARS Ambush! to operational units in Iraq within six months of 
learning of a motorized ambush training shortfall. In the attempt to use gaming 
technology to mitigate the damage to operational forces from ambush, DARPA 
built a tactical application on a civilian computer game called Operation 
Flashpoint (Peck, 2004). The game allowed soldiers to author their own 
scenarios on the fly. Soon, the Army used the simulation both in theater and 
stateside to enhance training. Additionally, personnel began to experiment with 
using the serious game for rehearsal with soldiers at Fort Polk, providing virtual 
tours of the places where troops would fight weeks later (Laurent, 2007). The 
project illustrated that not only could serious games provide training tools 
cheaply, but they could do it quickly. Just as important, soldiers liked the training 
and adapted to it readily. Roger Smith, the Chief Technology Officer for PEO- 
STRI, described soldiers using DARWARS Ambush!-. 

The soldiers just dive in and start ‘playing’ the scenarios. Then 
they start adapting those scenarios to make them more realistic. 

They are not only learning the given scenarios, but teaching 
themselves to replicate real-life experiences to re-live and recreate 
what they’ve seen on their own missions. (McLeroy, 2008) 

A 1996 conference, that brought the military, research, entertainment, and 
gaming communities together, spawned the creation of a modern-day RAND 
Corporation of sorts. The Institute for Creative Technologies (ICT) was initiated 
at the University of Southern California, in 1999, to pool the best ideas of all the 
communities into collaborative projects for the benefit of all. That same year, 
Michael Macedonia of PEO STRI proposed the development of a console-based 
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training system for the soldier on the ground. Five years later, ICT produced Full 
Spectrum Warrior. The game’s title references the Joint Vision 2020 concept of 
full-spectrum dominance, whereby military forces control situations ranging from 
relatively peaceful security operations to the worst of the nuclear threats. The 
game allows players to manipulate a squad through urban settings to destroy a 
variety of enemy. Full Spectrum Warrior demonstrates the type of partnership 
that serious games entail. The Army got a simulation, but the entertainment 
industry got a top-selling blockbuster when the civilian version hit the shelves of 
local stores. Full Spectrum Warrior has become a hit in the military, where it is 
even being considered for such nontraditional uses as re-creating traumatic 
contextual stimuli for treatment of PTSD patients (Halter, 2006). 

The final synthesis of leisure and war training occurred with America’s 
Army, a game to get civilians interested in joining the Army. The Army sought to 
capitalize on the relationship between the armchair teenage general and the 
individual soldier through the development of America’s Army. Touted as a 
recruiting tool, the concept of America’s Army extended well beyond luring youth 
to the recruiting office. The Army envisioned a tool that would attract the wired 
generation to the military through leisure activity. Then, the Army would use that 
same leisure activity to train the newly recruited soldiers on the job. Conceivably, 
a young man would play on America’s Army the week before his prom and train 
on the same platform months later in preparation for combat. America’s Army is 
unique in that it was developed completely inside the Department of Defense at 
the Naval Postgraduate School’s Modeling, Virtual Environments, and Simulation 
(MOVES) Institute. America’s Army introduced a new idea to the military training 
arena: gaming as a recruiting tool. It also represented the first government 
production of a video game in the public domain (Li, 2003). Traditionally, the first 
training a soldier or Marine saw was marching, shooting, running, and the 
discipline of a platoon sergeant. Now, the first training a soldier might see could 
be a serious game. The wargame had potentially gone from the pastime of a 
bored aristocrat to the initiating element in a soldier’s warfighting career. 
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Serious games offer another benefit to the military constantly on the go: 
deployability. The Marine Corps developed a program called Deployable Virtual 
Training Environment (DVTE) that combined a variety of personal computer- 
based simulations onto a single laptop. Suites of thirty of these laptops are being 
fielded to each infantry battalion in the active and reserve Marine Corps. The 
initiative turned simulation into something that extended beyond the physical 
bounds of a camp simulation center and beyond the reach of a technical 
contractor. DVTE will be readily available to all infantry units in their own training 
shops. With a moderately small number of computer cases, a deploying 
battalion can pack an entire simulation center along for the trip (Figure 1). Units 
can transform static forward operating bases and ship berthing into active 
training domains for the otherwise bored Marine. DVTE makes training available 
anywhere at any time. 



Figure 1. DVTE case for four computers and associated equipment 
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8 . 


The Military Does Not Know whether Serious Games Work 


This thesis has presented a series of serious games currently used by the 
military in the effort to demonstrate the proliferation of gaming technology in 
today’s forces. Examination of these examples shows that the games have 
come into use primarily because they are cheap, readily available, portable, and 
have incredible potential for training, rehearsal, and analysis. Most serious 
games are used because commanders instinctively recognized their potential 
and started using them. This process contrasts with the acquisition of most 
military systems in which the Department of Defense and contractors go through 
years of iterative design, testing, and evaluation before fielding a product. A 
critical review of Ed Halter’s review of serious games for military applications, 
entitled From Sun Tzu to Xbox, summed up the resulting problem well: “There 
just isn’t any evidence that any of it [serious gaming efforts in general] works, and 
Halter doesn’t even bother trying to prove otherwise” (Klein, 2006). This 
comment seems to extend beyond Ed Halter; little research has been done to 
determine the effectiveness of serious games. Consequently, these serious 
games are being used for serious purposes without a scientific background 
demonstrating their effectiveness. 

Consider the case of Marine Doom. Sergeant Snyder produced the game 
in the attempt to leverage technology for the benefit of the fighting Marine. While 
the game fulfilled stated training objectives, no one measured whether those 
objectives were truly met, and the game’s short-lived tenure casts doubt that they 
were. DARWARS Ambush! was produced and fielded within six months. 
Clearly, that time was spent on development rather than analysis of 
effectiveness. Even more fully supported programs like Full Spectrum Warrior 
and America’s Army have borne little scientific scrutiny into their effectiveness as 
training platforms. 

Today, there are not any generally accepted methods to evaluate the 
effectiveness of serious games (Roman & Brown, 2008). Moreover, there is no 
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accepted practice for how to use the serious games in training or analysis. A 
close look at what operators are actually doing indicates that serious games will 
never replace live training. Rather, they seem best suited as training multipliers. 
Serious games do not reduce live training; they make it more effective (Roman & 
Brown, 2008). Serious games seem most effective when used for the 
development of cognitive skills, such as decision making in a chaotic 
environment. Users do not learn psychomotor skills as well from serious games. 
A common problem in serious games is situational awareness. Serious gaming 
has not been without research; indeed, a growing body of work has shed some 
light on the topic. The second section of this chapter highlights some of this 
research that is relevant to this thesis. However, research thus far has only 
proven that serious games offer potential, but the benefit must be weighed 
against some important limitations. To explore these ideas more fully, the 
discussion will turn to the specific platform at the center of this research: VBS 
2™. Before leaving the more general discussion, a summary is in order. 

9. Our Stroll through History Tells Us Quite a Bit about Serious 
Games 

At this point, bulletizing key points from the preceding wargaming history 
is helpful: 

• Wargames have ties to war that are almost as long as the history of 
war itself. 

• Wargames have never replaced a training area in its entirety; they 
have supplemented training. 

• Serious games have potential to train at the tactical, operational, 
and strategic level. 

• Serious games apply in the cognitive decision-making domain. 

• Relative to live training, serious games can be a cheap, readily 
available form of training any time and any place. 
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• Serious games appeal to the new warfighter coming from the wired 
generation. 

• Serious games can tie the military to the civilian world and vice 
versa. 

10. VBS 2™ is the Marine Corps’ Bid to Capitaiize on the Potential 
of Serious Games 

The Marine Corps has moved forward with an initiative that illustrates the 
potential of serious games. With the relative success of Marine Doom, the 
Marine Corps sought a commercial gaming product that would enable tactical 
training across a wide range of warfighting domains. Bohemia Interactive, an 
Australian company, manufactured Operation Fiashpoint, a commercial game 
that could be modified to fit the service’s needs. The Marine Corps reviewed the 
game in depth and issued a statement of work to modify the game into a realistic 
Marine environment with exclusive rights for future modifications as necessary. 
VBS 2™ resulted from the request. VBS 2™ is a first-person shooter game that 
can be networked to involved many players at once. Using a drag-and-drop 
menu, game administrators can develop virtually any realistic scenario on the 
various terrain databases available in the system. Game versatility easily allows 
players to drive armed trucks through the desert, patrol dismounted in town, work 
with tanks, artillery, and aircraft in the open field, or interact on a limited basis 
with civilians and coalition partners. While various games may outperform VBS 
2™ in different areas, VBS 2™ offers the advantages discussed above to 
individual Marines at their personal computer or battalion laptop. 

However well VBS 2™ might look on the specification sheet, little has 
been done to demonstrate or prove that it actually works as a training device. To 
discuss this issue intelligently, one first must establish what VBS 2™ is actually 
supposed to do. The Marine Corps sought a virtual training platform for 
individual Marines for all the reasons simulations such as Fuii Spectrum Warrior 
were developed. Marines were already into gaming from their pre-service years. 
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serious games were cheap, serious games were deployable, serious games 
were easy to administer. The statement of work for VBS 2™ was based on a 
cognitive task analysis completed in 2004. Program Manager Training Systems 
(PM TRASYS) contracted with CHI Systems, Inc. and Klein Associates, Inc. to do 
something best explained by the report’s title “Using cognitive task analysis to 
support cognitive authenticity in training strategies for anti-terrorism force 
protection tactical decision making” (J. Styer, personal communication, October 
27, 2008, and P. Nichols, personal communication, November 12, 2008). 
Operation Flashpoint iu\i\\\e6 several goals quickly. It was the right size in terms 
of memory. It provided the right amount of versatility and fidelity. Its flexible and 
friendly mission editor was deemed critically important. Most importantly, it was 
close to what the Marine Corps needed, requiring the least amount of 
modification. As a result. Operation Flashpoint was the game of choice, and 
VBS 2™ was born. The only official document linking the game to performance 
standards was the PM TRASYS cognitive task analysis (D. Mathes, personal 
communication, October 27, 2008). 

The cognitive task analysis did specify performance criteria in measurable 
Marine Corps terms. At the time, the Marine Corps used a system of “Individual 
Training Standards” (ITS) that specified required tasks and the precise 
measurable steps to perform those tasks to standard. The cognitive task 
analysis listed the ITSs required for the missions they analyzed. The cognitive 
task analysis focused on the Fleet Anti-terrorism Security Team (FAST) 
company, an entity representing a relatively narrow, unique skill set. 
Generalizing this task list to the entire Marine Corps certainly shortchanges the 
capabilities of VBS 2™. The game can do so much more. Thus, identifying what 
VBS 2™ is actually supposed to do is muddy water, and the available 
documentation simply leaves one making things up based on intuition. 

VBS 2™ has experienced some scientific scrutiny, although it has come 
from outside the United States. In 2002, some Australian computer science 
researchers at the University of New South Wales began to explore Operation 
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Flashpoint as a tactical training tool. They evaluated users in a series of three 
exercises in the game. From survey results, they noted such problems as 
disorientation from the small screen size, lack of simulated fatigue, and 
unrealistic tactics among computer-generated forces. Players could not practice 
basic motor skills such as weapons drills. Nevertheless, they showed that the 
game could be used for tactical training with limitations (Barlow, Morrison, & 
Easton, 2002). Once the VBS project had matured somewhat, in 2004, the same 
Australian research team reviewed the game system again. They continued to 
rate the game as a good tool for section level training, although situation 
awareness issues still topped the list of limitations. The AAR capability 
requested by the Marine Corps added significant capability to the game 
(Morrison & Barlow, 2004). In the same year, one of the researchers worked 
with a group at the Virtual Environments & Simulation Laboratory (VESL) to study 
how squad leaders make decisions under the stress of battle. The team 
successfully demonstrated that serious games can be used for analysis as well 
as training (Barlow, Luck, Lewis, Ford, & Cox, 2004). The next year, the team 
highlighted a striking limitation of VBS 2™ and serious games in general: that is, 
players must practice with the game for periods ranging from a few hours, for 
basic skills, to a week or more for more complex team leading tasks in order to 
use the game seamlessly as an extension of their warfighting skills (Morrison, 
Barlow, Bethel, & Clothier, 2005). All of this research supported the murkiness of 
the situation: serious games have potential as a training tool, analytical asset, 
and rehearsal mechanism, but these benefits come with notable limitations. 
Clearly, the research suggested the need for a closer look at tactical 
performance on the ground after game training. 

More recently, the Canadian Combat Training Centre demonstrated some 
degree of effectiveness for VBS 2™. They incorporated the serious game into 
their Troop Warrant Officer’s course, reducing live field training while increasing 
performance results. Their experimentation resulted in very specific data 
showing that use of VBS 2™ resulted in a more effective blended training 
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program, but the effort was not structured in a way that would show specifically 
which aspects of the serious game worked and which did not. For one serial of 
the course, the school used 1 day of VBS 2™ training and 5.5 weeks of live 
training in the field. Noting success from use of the simulation, the school turned 
2.5 weeks of the live training into VBS 2™ training and noticed a significant 
improvement in success rate in field exercises. However, because the school 
was merely getting training done and not running a formal experiment, these 
results were tainted with confounds, such as changes in instructor cadre (Roman 
& Brown, 2008). 

The background study thus far has demonstrated the potential of serious 
games, and that VBS 2™ is well designed to realize that potential. While VBS 
2™ has been examined in the scientific and academic arenas, however, nothing 
but anecdotal evidence exists to demonstrate its effectiveness. Intuition and 
research thus far indicate that VBS 2™ has much to offer the individual Marine. 
Clearly, a glimpse into this effectiveness, or lack thereof, is warranted. The next 
section examines how this might be accomplished. 

C. TRAINING TRANSFER 

Training transfer is the idea that training produces results in the form of 
trainee performance improvement as intended. It seems simple enough, on the 
surface, to test: just get two groups of people, train one group the old-fashioned 
way and the other group using the method under investigation, and compare the 
results. Intuition dictates a few complexities, such as the need for measurable 
results and the standardization of peripheral factors so that the effects of the new 
training method can be isolated. However, training transfer studies are not 
necessarily as straightforward as they may seem, and an examination of past 
studies and related research sheds some light on the difficulties. 

Several combinations of words exist in conjunction with the word “transfer” 
that revolve around a similar group of concepts. For the purposes of this thesis, 
it is worth taking a closer look at two of these word groupings. Transfer of 
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training, which is assumed to be the same as training transfer, is defined as “the 
extent of retention and application of knowledge, skills, and attitudes from the 
training environment to the workplace environment” (Bossard, Kermarrec, Buche, 
& Tisseau, 2008). Transfer of learning is a slightly different concept that applies 
to the educational arena. If transfer of learning is accomplished, a student can 
generalize something learned in the classroom to broader application areas, so 
that the individual uses knowledge gained from the learning process to solve 
problems that have not been presented before. As a rule of thumb, transfer of 
training generally applies to work and transfer of learning applies to education 
(Bossard et al., 2008). Both concepts share the idea that knowledge and skills 
can be applied in a practical, measurable environment. 

In virtual environments, a concept appears throughout relevant literature 
that differentiates between transfer related directly to the training context and 
transfer generalized beyond the context. Vertical transfer refers to the ability of 
the learner to recognize elements of the training context and apply what he or 
she has learned to problems of increasing complexity. Horizontal transfer, on the 
other hand, describes the learner’s application of knowledge and skills gained 
through the training to general problems that extend beyond the immediate 
context of what was taught. Other terms, such as “near” and “far” or “general” 
and “specific” have slightly different meanings but hinge on the same idea. 
Whatever the terminology, it is important to keep in mind that transfer of largely 
procedural knowledge in a set list of clearly defined steps is much different than 
transfer of generalized situational knowledge applied in varying circumstances. 
Moreover, context is considered critical; some argue that context is so important 
that straying from the original context eliminates the possibility of transfer 
(Bossard et al., 2008). 

1. The Trainee Matters 

People learn in different ways, and these differences affect the level of 
transfer. Cognitive ability affects the amount of transfer, with one study 
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concluding that this factor accounts for as much as 16% of the variance in 
training effectiveness. Self-efficacy has its own effect; if an individual does not 
feel competent to perform a task, the training transfer will quite likely suffer. 
Motivation can affect transfer in a couple of ways. An individual may or may not 
be motivated to train, influencing the effectiveness of the effort. An individual 
may not be motivated to use the skills in the real world, an issue called 
motivation to transfer. Personality traits can affect transfer; among those studied 
with transfer impact are anxiety, openness to experience, extroversion, and 
conscientiousness. Training transfer tends to be maximized when the individual 
perceives that the knowledge and skills being trained will improve a relevant 
aspect of his or her work performance. Training transfers better with increases in 
the trainee’s job involvement. Training transfer also depends on the degree to 
which the individual identifies with his job and considers improvements in 
performance important to his self worth. Science has not yet shown the perfect 
profile of the individual who will easily transfer training or the individual who will 
struggle with it, but the science is clear that transfer will differ from person to 
person. Some key characteristics, such as cognitive ability, pretraining 
motivation, negative affectivity, and perceived utility, may help identify those 
individuals who will struggle in a training transfer endeavor. More importantly, 
one may be able to take actions in the training phase to minimize the effects of 
some of these characteristics, such as negative affectivity or perceived utility 
(Burke & Hutchins, 2007). 

This thesis does not seek to expound in depth on the various schools of 
thought in learning psychology, but it is helpful to quickly highlight the three most 
common theories: behavioral learning, cognitive learning, and constructive 
learning. Psychologists adhering to the behavioral approach contend that the 
human mind is a sort of black box, where inputs can be studied and outputs can 
be studied, but the mental processes in the middle are ignored. Key side notes 
from this school of thought are the idea that the learner adapts to the 
environment and the concept of learning as a largely passive process. 


38 



Proponents of cognitive learning, on the other hand, focus on the black box itself, 
attempting to model what goes on inside the learner’s brain. In cognitive 
learning, the individual conceptually has a knowledge base and uses processes 
and symbols to map what is taught into the base. Thus, study of this approach 
seeks to identify and describe those processes and symbols. Both behavioral 
and cognitive learning approaches view the learning process as one in which the 
knowledge presented is set and absolute, like a traditional classroom lecture 
environment. Psychologists adhering to the constructivist school of thought 
challenge this idea, contending that each learner reconstructs the facts 
presented in his or her own way to build the internal knowledge base. Like the 
cognitive approach, knowledge is viewed as a combination of symbols and 
processes to access a knowledge base in the head, but the constructivist 
approach holds that each individual has his or her own world perspective based 
on individual experience (Bossard et al., 2008). 

Military training often reflects a behavioral approach out of necessity. 
Training typically occurs en masse, and proficiency is achieved through 
repetition, discipline, and remediation. The situation changes when training 
orients on decision making. In a platoon training event, a junior Marine may 
experience a primarily behavioral training regimen while the platoon commander 
experiences a more cognitive or constructivist approach. As pointed out in the 
training transfer introductory section, transfer itself may be contextually specific 
or more generalized. These taxonomies become important to the discussion of 
unit level training. In the team environment, some individuals may be learning 
very lock-step, procedural tasks while others may be learning very abstract, 
generalized decision-making schemes that will most likely be applied in new and 
unique situations. When the overall transfer of group training is under the 
microscope, one must bear in mind the different training models in place at the 
individual level. 

When technology is used for training, the trainee’s attitude toward that 
technology can influence the training experience. A U.S. Army project explored 
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this idea. A first-year class of cadets at the U.S. Military Academy trained with 
America’s Army, a game discussed in the first section of this chapter. The 
research team sought to determine whether gaming experience and computer 
self-efficacy affected four learner outcomes of interest: training motivation, 
training satisfaction, ease in using the game interface, and perceived team 
cohesion. Not only did the research team find that these two factors affected 
learner outcomes, but they found that the type of experience mattered as well. 
For example, gamers who used products similar to America’s Army had higher 
training motivation, training satisfaction, ease in using the interface, and 
perceived team cohesion than gamers who played much different games (Orvis, 
Orvis, Belanich, & Mullin, 2005). As pointed out above, a trainee’s learning style 
matters, but a trainee’s experience with computer technology in general, and 
experience with technology similar to that used for training, makes a difference 
as well. 

2. The Way Training is Done Matters 

The design and delivery of the training intervention can affect transfer. 
Before the training is planned, a needs analysis can determine whether it is 
needed. If the training is not necessary, training transfer results will obviously 
suffer accordingly. During the training intervention, establishment of learning 
goals and objectives positively impacts training transfer; that is, people learn 
better if they know what is expected of them in the course of the training. 
Content relevance is also important; trainees must see a relationship between 
training content and work tasks. 

The way the material is taught influences training transfer. Practice and 
feedback provide the opportunity for trainees to reinforce what has been 
presented, enhancing transfer in the long run. Overlearning, the practice of 
having individuals continue practicing even after they have correctly 
demonstrated the skill, can improve transfer of training by making responses 
automatic. Trainees can experience cognitive overload, in which they are 
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presented more than they can learn at one time. Strategies that minimize 
information not necessary for learning, while maximizing the information that 
directly contributes to learning, improve training transfer. Active learning quite 
likely affects transfer more positively than lecture, although the literature is 
inconclusive on this point. Error-based examples help trainees learn a skill by 
showing aspects of the task that can go wrong, and such strategies have been 
found to increase training transfer (Bossard et al., 2008). 

A common workplace phenomenon is the individual returning from a 
training workshop with the newest way of doing something. Quite likely, that 
individual returns to a skeptical crowd, unwilling to bother with changes to the 
status quo. In the end, the individual’s training might end up useless. The 
individual may have learned the knowledge and skills perfectly. He or she may 
have been able to apply them any time and any place. The individual may have 
returned with a burning enthusiasm to apply the knowledge and skills 
immediately. Nevertheless, all of this failed in the face of resistance back in the 
office. 

The organization itself is critical to training transfer. For example, training 
transfer improves if the trainee perceives a direct link between the training and 
the supervisor’s strategic goals. The propensity for the organizational 
atmosphere to contribute to training transfer is described by the term “transfer 
climate.” A positive transfer climate encourages use of the newly trained skills 
and incentivizes their correct application. Supervisory support impacts training 
transfer. The support of peers is even more important than the support of 
supervisors. If a skill and applied training technique gain acceptance at the water 
cooler, transfer improves. The organization also contributes to positive training 
transfer by affording the opportunity to perform the newly acquired skills. The 
organization’s methods and pressure to hold individuals accountable to what they 
have learned seems to affect training transfer as well, although the available 
research fails to describe this link in detail (Bossard et al., 2008). 
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3. 


Numbers Matter 


The practical matter of assessing training transfer is based on quantifying 
results of some job or skill performance. Ultimately, performance must be 
translated into a number or combination of numbers that can be statistically 
compared and analyzed. Performance measurement is a science all of its own, 
and the problems are exacerbated when reviewing performance of teams instead 
of performance of individuals. Jack Zigon presides over a consultant firm 
specializing in team performance measurement systems. He highlighted three 
reasons team performance measurement is difficult: 

• It is not always obvious what results should be measured. 

• Even if you know what to measure, it is often not clear how the 
measurement should be done. 

• Teams are made up of individuals, thus measurement must be 
done at both the team and individual levels. 

Zigon’s work oriented on the corporate world, but he offered some advice 
that is useful for any performance measurement. First, he recommended 
focusing on results, because data related to the activity that produced the results 
can be misleading or uninformative. Second, he discussed the creation of 
measures for each accomplishment, noting that measures cannot always be 
numerical. He categorized measures into numeric measures that use 
quantification of some concrete aspect of the accomplishment and descriptive 
measures that use words for evaluation. Third, he recommended developing a 
system of performance standards. These standards use the measures from the 
second step to determine how a team performs against a set goal. Finally, Zigon 
suggested the implementation of a feedback system. The feedback system 
provides the individual, the team, and the organization an assessment of the 
performance (Zigon, 1998). 

The project that is the subject of this thesis seeks to capitalize on years of 
Marine Corps experience in team performance evaluation to prevent a 
reinvention of the wheel. The Marine Corps Mission Performance Standards 
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(MPSs) list all of the skills Marines in a particular job specialty should perform to 
be proficient within that specialty at each rank of service. Marine Corps Order 
(MCO) P3500.72A, dated 18 April 2005, outlines the training and readiness 
program that incorporates these MPSs into a training regimen for ground forces. 
In the publication’s introduction to Marine Corps training philosophy, MCO 
P3500.72A explains in very direct terms how the service views training and 
training evaluation: “Training Marines to perform as a team in combat lies at the 
heart of the Training and Readiness (T&R) Program. Unit readiness and 
individual readiness are directly related.” The publication then specifies the 
tenets on which the T&R concept is built: 

• Focus on expected combat missions 

• Building block approach to training 

• Focus on Individual Core Skills and Unit Core Capabilities 

• Organization of tasks into executable events 

• Sustainment of training 

MCO P3500.72A addresses core skills, which apply to the individual, and 
core capabilities, which apply to the section or unit. An individual must be able to 
perform core skills to be qualified for his or her job specialty. A section or unit 
must be able to perform core capabilities in order to meet performance 
expectations in contingency operations or combat. Closely related to these 
concepts are Mission Essential Tasks (METs) which are those tasks that form 
“the very essence of the community’s existence.” Skills that are “environment, 
mission, rank, or billet specific” are core plus skills, and advanced functions that 
are “environment, mission, or theater specific” are core plus capabilities. In order 
to understand the organization of the MPSs, one must keep in mind the building 
block approach used by the training and readiness program. The program 
organizes the standards as executable events that units can arrange into 
appropriate field exercises and training programs. Some standards serve as 
prerequisites for others, and units can use chaining to give sustainment credit for 
simpler tasks that support more advanced events. In this way, the training and 
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readiness program serves as the basis for school syllabuses as well as 
peacetime training in operational units. The same program that defines 
requirements for entry-level job specialty training also specifies how individuals in 
that job specialty will train throughout their careers and how their units will train 
as well. 

The training and readiness program evaluates training in two ways. 
Proficiency measures performance of a certain skill or set of skills against a set 
standard. Currency evaluates the standard against a sustainment interval for the 
particular event. Thus, training may be necessary because an individual or unit 
cannot perform a skill well enough, or because too much time has elapsed since 
the skill was last performed. 

The administration of the training and readiness program supports 
organizing and analyzing training events. Each MPS is coded with a series of 
three four-character codes. The first four-character grouping indicates the job 
specialty, the second four-character grouping indicates the functional area or 
duty area, and the third four-character grouping indicates the level and 
sequence. The program ultimately seeks to measure a unit’s preparation for 
each of its METs. The Combat Readiness Percentage (GRP) quantifies this 
preparation. Specific events that support a MET for GRP calculation are called 
evaluation coded events or E-coded events. Thus, E-coded events are those 
that contribute to an overall grade or numerical evaluation. 

Each MPS has a title and a description that explains the purpose, 

objectives, goals, and requirements of the event. The event’s condition lists the 

items, such as equipment, manuals, tools, and aids, that must be provided as 

well as any specific conditions under which the event must be performed. The 

event’s standard explains the minimum acceptable level of performance of the 

event and how the level of performance will be judged. Each event has a list of 

performance steps that guide an evaluator to ensure all components of the MPS 

have been satisfied. An event may have a list of prerequisites or chained events. 

Finally, each event has a list of references and support requirements. For each 
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task supporting a unit’s mission, the training and readiness system specifies what 
the task is, how to do it, how to measure it, and how to support it. 

Navy and Marine Corps Publication (NAVMC) 3500.87 is the Marine 
Corps’ infantry training and readiness manual. With more than 600 pages of 
documentation, the publication explicitly states how the Marine Corps infantry 
should train, from the individual riflemen through the large battalion and 
regimental staffs. NAVMC 3500.87 meticulously follows the guidelines of the 
training and readiness program, providing a highly detailed and organized 
method of delineating what tasks must be accomplished for any given mission, 
how they will be accomplished, and how they can be graded. 

4. Determining What Matters 

Lisa Burke and Holly Hutchins (2007) collaborated on an integrative 
literature review of training transfer. In summarizing available research, they 
recommended that training transfer be assessed as “a multidimensional 
phenomenon with multilevel influences.” This concept is probably the most 
critical idea related to training transfer. Transfer is not a black and white switch 
that either happens or fails to occur. Rather, the individual trainee, the training 
plan, and the training environment all have many variables working together to 
influence the degree of transfer. Understanding these potential variables allows 
one to try to minimize their impact but, in the end, many factors can affect the 
final transfer of training. 

Applicable literature recognizes this multi-factored aspect of training 
transfer. One model proposed to deal with the issue is the generalized Learning 
Transfer System Inventory. LTSI attempted to design a scaling system for 
measuring transfer based on various factors. The generalized study included 
sixteen factors analyzed via questionnaire. While LTSI took individual level 
training transfer a step too far for this thesis, an overview of the sixteen factors is 
useful. Some of the factors focused on the individual, such as learner readiness, 
motivation to transfer, positive and negative personal outcomes, and personal 
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capacity for transfer. Other factors focused on the environment or climate such 
as peer support, supervisor support, supervisor sanctions, performance 
coaching, and opportunity to use. Several factors related to the way the 
individual perceives the training affecting him or her, such as perceived content 
validity, performance expectations, performance outcomes expectations, and 
performance self-efficacy. Finally, the transfer design itself affects transfer. 
When researchers study training transfer, reporting information about the 
circumstances of the training experiment using these factors can help 
standardize studies in the aggregate. More practically, though, the LTSI model 
itself can help the training organization as an indicator of transfer inhibitors so 
that they can be minimized. For the purposes of a training transfer study like the 
one proposed in this thesis, LTSI provided a sort of guideline for factors that 
need to be controlled throughout the study in order to isolate the effects of the 
training mechanism itself (Flolton, Bates, & Ruona, 2000). 

In summary, training transfer depends on many factors, and any attempt 
to measure transfer must take this into account. A list of pertinent notes from this 
study follows: 

• Trainees learn differently. 

• Transfer improves if the trainees know what they are doing and 
why. 

• The training climate can positively or negatively impact training. 

While the Marine Corps has a ready-made system of evaluating training, these 
factors proved important throughout the course of the project. 

5. A Tribute to Those Who Have Gone Before 

Simulator use without robust training transfer research is prevalent across 
a wide variety of training domains, including medical procedures training, nuclear 
power plant operation, commercial aviation, and the NASA space program. The 
bottom line is that training transfer research is hard. Fluman performance is 
difficult to measure, and the difficulties expand when the exercise involves 


46 



teams. Additionally, training is quite likely not the sole contributor to subsequent 
proficiency. A proper evaluation must isolate training from other factors such as 
environment, leadership, and equipment (Thurman & Dunlap, 1999). 
Nevertheless, some research projects have evaluated the training transfer of 
personal computer-based training. While the following list does not strive to be 
comprehensive, it demonstrates some of the efforts that have been made and 
the successes and challenges encountered along the way. 

A research project at the Israeli Air Force flight school used a personal 
computer-based game called Space Fortress II to help train cadet pilots. Two 
groups of cadets received ten hours of simulation-based training while a control 
group received traditional training. The effects of the training were evaluated in a 
series of eight flights that were part of the training program. Flight instructors 
graded the cadets’ performance on several criteria with numerical markings and 
then gave the students an overall score. Comparison of the results indicated that 
both simulation groups performed better in the live flight exercises than the 
control group. One of the simulation groups trained using single task games. 
These part-task trainees trained with one part-task game at a time with feedback 
on each game. Afterward, the trainees played the whole game. The other group 
trained with the whole game only, receiving feedback at the end. The 
researchers found that the trainees who had used the part-task games 
outperformed the cadets who trained with the full game (Gopher, Weil, & 
Bareket, 1994). This study shows how a personal computer-based game can 
result in enhanced performance in the real world. 

In 1998, a British study explored the use of Microsoft Flight Simulator to 
train pilots. This study featured two simulation groups: one group completed the 
simulation using controls similar to those in an airplane, while the other group 
used keyboard and mouse controls. A control group trained using traditional 
means. Students then flew real-world flights after the training, and flight 
instructors rated designated skills on five-point scales. The study determined 
that both simulation-trained groups performed better than the control group. 
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Also, the group trained in the simulation with controls that were similar to those in 
an airplane performed better than those who used the keyboard and mouse 
interface. The researchers concluded that a low resolution trainer such as a 
personal computer-based game could effectively contribute to training in the 
early stages of a pilot’s career. The authors were careful to point out that their 
results did not suggest that personal computer-based simulations aided in 
psychomotor skill acquisition. Also, the study was limited to a relatively simple 
flight task of flying a square where subtasks included flying straight and level and 
entering and exiting turns. The study could not generalize its findings to more 
complex tasks like taking off and landing (Dennis & Harris, 1998). However, this 
study shows another definitive instance of personal computer-based training 
enhancing real-world performance. 

A 2007 study at Bristow Academy, with a commercial personal computer- 
based game called X-Plane, sought to determine whether personal computer- 
based gaming technology could be used for helicopter training. The experiment 
involved three groups, differing by interface: those in a mock cabin with motion 
feedback, those in a mock cabin without motion feedback, and those using a 
desktop interface. The researchers measured training transfer by determining 
whether each group showed improvement in the simulator. Analysis of 
experiment results yielded conclusions that both cabin arrangements provided 
training transfer while desktop configurations resulted in no significant transfer. 
Also, experiment results indicated that motion did not significantly contribute to 
training transfer, although feedback questionnaires from participants indicated a 
strong desire for motion feedback. A deeper look at the difference between 
cabin and desktop interfaces suggested that the real contributor may have been 
visual feedback, because the cabin screens were larger than desktop monitors. 
Interestingly, the school that served as the test site opted for a full mission 
simulator instead of the game approach used in this research, despite the 
experiment’s indications that training effectiveness could be gleaned from this 
“serious game.” However, this study was somewhat limited in scope. In this 
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study, only the simulator performance provided evidence of training 
effectiveness; there was no attempt to see whether performance in the actual 
aircraft improved (Proctor, Bauer, & Lucario, 2007). 

Other training transfer experiments explored the use of personal 
computer-based training outside the aviation training community. In 2005, two 
Naval Postgraduate School students built a first-person shooter simulation to 
train for artillery forward observer procedures. Forward Observer Personal 
Computer Simulation (FOPCSIM) allowed a single user to learn basic call for fire 
procedures and practice them in a realistic virtual environment. The project then 
took the simulation to the Marine Corps basic officer training course where call 
for fire is one of the fundamental skills taught. They used FOPCSIM to train one 
group of Lieutenants, and the other Lieutenants were trained with the old theater 
simulation called Training Set, Fire Observation (TSFO) where call for fire results 
were projected in a slide show. The research used the basic school written exam 
as a metric for determining proficiency. The researchers determined that the 
personal computer-based game performed as well or better than the TSFO. 
Most notably, FOPCSIM was much more efficient; since trainees were 
automatically scored, they could perform as many missions as their pace would 
allow, instead of proceeding at the pace of instructor observation in TSFO. 
Flowever, the research project was unable to test the effects of the training in live 
fire (McDonough & Strom, 2005). 

The training transfer examples cited so far focus on procedural trainers for 
individual tasks. This thesis seeks to explore the domain of small unit tactical 
operations, and some research has investigated the effectiveness of personal 
computer games in this area. In 2005, a separate Naval Postgraduate School 
project sought to determine whether commercial off-the-shelf (COTS) gaming 
technology could improve training. U.S. Army Majors J. Nolan and J. Jones used 
Delta Force: Black Hawk Down-Team Sabre™ as a small unit tactics training 
platform at the Infantry Officer Basic Course in Fort Benning, GA. The two 
researchers compared survey data from a control group who did not use 
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simulation and a test group who used the COTS game. They found only limited 
statistical differences in the soldiers’ confidence across a 50-question survey. 
Primarily, the surveys indicated that the simulation training did not hinder 
performance in any way, but it provided virtually no evidence that the simulation 
improved performance. The researchers described the most interesting data as 
group interview results where the officer controlling the exercise noticed 
improvements in movement in tactical formations as well as better use of 
binoculars in security tasks. Nolan and Jones’s study demonstrated that serious 
games could be used for training, showed no evidence that the training technique 
was better than traditional means (Nolan & Jones, 2005). 

Starting in 2000, the Office of Naval Research sponsored a large research 
project called VIRtual Technologies and Environments (VIRTE). This program 
studied the use of simulation and virtual environments in many different ways to 
support the emerging over-the-horizon amphibious triad of the new landing craft, 
amphibious personnel carrier, and tilt-rotor aircraft. The project emphasized 
training transfer as the ultimate objective from the very beginning (Muller, Cohn, 
& Nicholson, 2003). Part of the project involved the development and evaluation 
of a personal computer-based game called Combined Arms Network (CAN) to 
train Fire Support Teams (FiSTs). The resulting transfer study was one of the 
most comprehensive personal computer-based game transfer studies ever done. 
Experiments were conducted with all facets of the Marine Corps, including active 
and reserve units, operational units and schoolhouses, basic training and 
advanced training, and ground and amphibious operations. The research project 
involved the administration of surveys and, when possible, the evaluation of live 
exercises after training. In general, the research showed evidence for training 
transfer in all groups through increased efficacy ratings on the surveys. This 
project was unique because its breadth exposed the individual appeal of a given 
training tool. That is, efficacy increased, but it increased for different reasons for 
each group studied (Becker et al., 2009). While this study was an overall training 
transfer success, it points to the dangers of over-generalizing results. 
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Another recent transfer study endeavored to show the effectiveness of 
serious games for unit training. A 2007 study used a personal computer-based 
game called Close Combat: First to Fight to train urban combat operations. In 
this experiment, a sixteen-Marine simulation group used the game for training 
while another sixteen-Marine group used traditional training methods. 
Performance was evaluated in a live fire shoot house, where the Marines 
conducted a standardized Close Quarters Battle (CQB) exercise in four-man 
teams. Also, Marines completed a survey in which they rated the usefulness of 
the simulation training for a list of tasks that were trained in the exercise. 
Statistical analysis showed no difference in performance in the live exercise as 
assessed by the objective grading of the CQB. However, the survey results 
indicated that Marines felt the simulation training increased their proficiency in 
the tactical skills that were trained (Proctor & Woodman, 2007). 

6. The Road Ahead 

After looking at these training transfer studies as a group, one can note 
some important trends: 

• Research has shown most success in the training transfer arena 
when focused on individual, procedural tasks. 

• Three measures of effectiveness are commonly used: surveys to 
gain insight into participants’ self-assessed proficiency, knowledge 
tests to determine participants’ academic understanding of the 
skills; and instructor- or Subject Matter Expert-rated evaluation in 
the real world. 

• Qf the three measures of effectiveness listed above, the subjective 
self-assessed proficiency is most likely to show results, particularly 
for collective, cognitive tasks like a small unit tactical exercise. 

• Personal computer-based game effectiveness is more apparent 
when the research project involves novices. 

• Personal computer-based games can produce enhanced 
performance in the real world, but research has not yet established 
a pattern or formula for when they are effective and when they are 
not. 
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Training transfer in other simulation domains has drawn attention because 
of its difficulty, so researchers have studied groups of transfer projects to 
determine Best Practices and pitfalls. Boldovici (1987) compiled one of the 
earliest examinations of modern simulation transfer studies. He listed the 
following pitfalls: 

• Small numbers of soldiers or crews are used in the comparison; 

• Subjects in the compared groups are not matched or randomly 
assigned; 

• Groups are treated differently in respects other than those under 
investigation; 

• Weapon system error masks training effects; 

• Amount of practice is insufficient to affect proficiency; 

• Ceiling or floor effects mask differences between groups; 

• Measurement of Task B performance is unreliable; 

• Inappropriate analyses are used to estimate transfer (Boldovici, 
1987). 

A component of learning theory that is particularly pertinent to the use of 
personal computer-based games for training is experiential learning. According 
to this theory, people learn from the experience of the task. Recent research in 
this topic describes experiential learning as a cycle of experiencing the 
environment, observing behavior and reflecting on the experience, generalizing 
based on the reflection, and modifying concepts based on new experiences. The 
researchers propose an idea that is interesting to military trainers: trainers can 
tailor the environment so that learners incur certain experiences according to a 
structured plan than can flex to the needs of the trainee. In order to accomplish 
such a plan, the learning events must: 

• Engage the learner mentally. 

• Emulate real-world requirements. Real-world refers to the physical 
environment and the cognitive tasks. 

• Allow the learner to experience effects of decisions. 
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• Require learner to reflect on outcomes of their actions. Build on 
established military practices of debriefs, lessons learned, and after 
action reports. 

• Revisit experiences increasing complexity of experiences to expand 
learners’ knowledge and skills by increasing number of events, 
pacing and emotional intensity (Menaker, Coleman, Collins, & 
Murawski, 2006). 

Personal computer-based learning is a tool for experiential learning, and the 
list above provides a formula for what the serious game and the trainer must 
accomplish together in order for the trainee to learn. While this formula does not 
guarantee training transfer by any means, it provides guidance on how to structure 
the training event to create the best environment for training transfer to take place. 
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III. PREPARATORY EXPERIMENTS 


A. INTERFACE FAMILIARIZATION PILOT STUDY 

1. Introduction 

Serious games face a potential drawback because of their interfaces. 
Game manufacturers expend much effort producing the most intuitive, efficient, 
and user-friendly interface for their software but, for any game, users must learn 
the interface. Military personnel training with a serious game must be able to 
move, shoot, and communicate as they would in live training. The game 
interface must serve as an extension of the servicemember’s warfighting skills. 
Interface training clouds the potential gain of serious game training because of 
the extra time required. If the individual did not have to learn the game, he or 
she could do something more productive. The military does not need gamers for 
gaming’s sake; gaming only serves as a means to an end. With this in mind, 
military serious game endeavors should strive to minimize time lost to learning 
the interface. 

This section summarizes a pilot study conducted as part of a project to 
examine the training effectiveness of Bohemia Interactive’s Virtual Battlespace 
2™ {VBS 2™) to train small unit tactics in a personal computer-based 
environment. In order for the simulation to be useful. Marines must use the 
interface to proficiently drive vehicles, shoot weapons, and maneuver their 
bodies as they would in the real world. We sought to provide the minimal 
computer interface skills to enable Marines to operate in a small unit mounted or 
dismounted environment. We developed an interface training program to 
prepare Marines as quickly as possible so that they could devote most of their 
simulation time to tactics training. 

The military seeks serious games to support training for a variety of 
reasons including their deployability, relatively low cost, and flexible training 
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environments. Additionally, the military wants to keep in touch with the wired 
generation, the youth who have experienced a childhood full of MP3 players, 
compact discs, computers, cell phones, and XBoxes. In 2008, a market research 
firm called the NPD Group reported that 72% of the U.S. population played video 
games in 2007 (Antonucci, 2008). Later in the year, the Pew Internet & 
American Life Project found that 97% of 12- to 17-year-old respondents played 
video games, they played them often, and they played a wide variety of games 
(Irvine, 2008). However, the military cannot count on gaming skills in all of its 
personnel. Roger Smith serves as the Chief Technology Officer for the U.S. 
Army Program Executive Office for Simulation, Training, and Instrumentation 
(PEO-STRI), so, in looser language, he is the Army’s official shopper for serious 
games. In August 2008, Smith responded to an interview question about modern 
soldiers’ familiarity with computer games: 

Our research and hands-on experience shows that about 50% of 
young enlisted soldiers call themselves “gamers” or are familiar 
with the mechanics of game play. At the officer level it is around 
33%. We have learned that we cannot assume that all soldiers 
have this familiarity. (Atkinson-Bonasio, 2008) 

A U.S. Army research effort to determine the influence of gaming 
experience on trainee satisfaction with serious games showed similar results. A 
survey of the first year class at the U.S. Military Academy revealed that 17% of 
the cadets had no gaming experience and 44% had limited gaming experience. 
The researchers concluded that an orientation with relevant games would likely 
enhance the training experience (Orvis et al., 2005). While the wired generation 
may know cell phones and iPods, they do not necessarily know how to use the 
games that support military training. Moreover, military training cannot leave the 
50%, 28%, 3%, or any other percentage of non-gamers behind. To use serious 
games for tactical training, all servicemembers must play. The experience of the 
wired generation helps, but the military simulation professional must craft 
exercises with the novice user in mind. 
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VBS 2™ has enjoyed some scientific scrutiny, and this prior research has 
shed some light on the time required to use the simulation as an extension of 
one’s warfighting skills. In 2004, an Aussie research team conducted a week- 
long trial to determine the potential utility of VBS 2™’s predecessor VBS 1. The 
trials involved a group of participants with varying degrees of computer and 
gaming experience with roughly half the people having no gaming experience. 
The study determined that nearly 80% of new users can attain individual skill 
proficiency within a couple hours. For higher-level cognitive skills such as 
situational awareness and team leadership, people need up to two days to 
become proficient in the game play (Morrison et al., 2005). The author’s 
personal experience with the simulation at the I Marine Expeditionary Force 
(MEF) Simulation Center in Camp Pendleton suggested one could reduce this 
timeline. There, Marines started training with only a 30-minute brief, and a 
telephone call confirmed that the I MEF Simulation Center currently uses this 
practice (D. Gerdes, personal communication, January 14, 2009). The I MEF 
Simulation Center uses a locally crafted slide presentation to brief Marines for 
about 15 minutes. They provide about 15 minutes of free practice time and 
supply users with a single page “cheat sheet” of interface commands. Flowever, 
the I MEF Simulation Center has many Marines trained to help simulation users 
during the conduct of the exercise. This pilot study sought to capitalize on this 
model and develop an interface training program to prepare VBS 2™ users within 
the 30-minute goal. 

As a pilot study supporting a larger project, this project served two main 
purposes. First, as the background literature indicates, progress in future VBS 
2™ projects requires knowledge of how to handle the interface. One cannot 
determine how well VBS 2™ contributed to some aspect of training if a user’s 
confusion with the interface muddies the waters. Thus, this project started with a 
guess of an appropriate technique for accomplishing interface training and 
sought to determine whether the technique was adequate. 
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Second, for planned future projects, groups and teams will use VBS 2™ 
as a common environment to interact toward some unit goal. A lone researcher 
in a laboratory can troubleshoot only a limited number of issues in support of 
such a project. Common sense dictates that many unpredicted problems and 
quirks will arise from group use that the lone laboratory developer can never find 
by himself. In this regard, this pilot study served as the first effort in the larger 
project of having multiple people work together in a common environment. 

This pilot study shed light on the answers to two questions: 

• Will the 30-minute presentation and practical application technique 
of interface training suffice to enable users to practice small unit 
tactical skills in VBS 2™? 

• What unanticipated problems will arise when a group operates in a 
common VBS 2™ environment? 

The pilot study supports a larger project to use VBS 2™ to train Marines in 
small unit convoy tactics. That project involves Marines patrolling a designated 
course in six to eight vehicles armed with heavy weapons. The course trains the 
following skills: react to an unexploded improvised explosive device (lED), react 
to an lED detonation, take immediate action against a blocked ambush during a 
convoy, take immediate action against an unblocked ambush during a convoy, 
evacuate a damaged vehicle, and evacuate a casualty. From these skills, we 
derived a list of VBS 2™ interface skills a user must know to use the simulation 
for the convoy training. These interface skills include: individual body 
movements, manipulation of personal gear, use of personal weapons, vehicle 
interaction and operation, use of vehicle weapons, manipulating a casualty, 
towing a vehicle, disarming an lED, and recognition of friendly, civilian, and 
enemy avatars. The pilot study endeavored to train these interface skills. 

To answer the first question listed above, we developed a criterion for 
success. We based success on the level of comfort users felt in performing each 
of the tactical skills listed above. We determined this comfort level through Likert 
scaled responses to each of the skills. We viewed success as a mid-level 
comfort rating for all skills. 
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2 . 


Method 


a. Participants 

Twelve Naval Postgraduate School Modeling, Virtual Environment, 
and Simulation (MOVES) graduate students, ranging in age from 27 to 41 years, 
participated in the pilot study. All participants volunteered as part of a seminar 
course on current simulation technology. All the participants were male military 
officers with service times ranging from 4 to 24 years. Four foreign officers 
participated, and U.S. officers represented all four of the armed services. 

b. Apparatus 

The project used twelve Dell Precision M6300 laptops from a suite 
of the Marine Corps Deployable Virtual Training Environment (DVTE) package. 
Peripheral equipment, including mice, cables, and switches, came from the 
standard DVTE package. For the familiarization training and evaluation, we 
networked the computers in pairs with four computers per switch. As shown in 
Figure 2, paired users faced each other at a classroom table so that they could 
not see each other’s screens. For the follow-up evaluation, all twelve computers 
were networked together in the same environment using two switches with six 
computers per switch. Participants operated as individuals in the follow-up 
evaluation. 
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Figure 2. Classroom setup for 12 participants 

c. Procedure 

This project used the I MEF Simulation Center’s training model of a 
slide presentation with practice time. The project also provided a single page 
cheat sheet of interface commands, included in Appendix E. The project 
modified the interface training by having the trainees follow along in the 
simulation on their own DVTE computers during the slide presentation. The I 
MEF Simulation Center cannot use this method because of the physical layout of 
the facility, but we assumed that the hands-on application would enhance the 
interface training. With this construct in mind, the project developed a 
familiarization scenario to guide novice users through the basic individual, 
weapon, and vehicle skills necessary to use VBS 2™. 
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To support the development of this scenario, the project started by 
identifying the specific interface skills needed for future tactical training. The pilot 
study supports a larger project oriented on tactical convoy training. The Marine 
Corps convoy training supports well-documented service training objectives, and 
analysis of these training objectives clarified the interface skills needed for the 
pilot study. An individual must be able to maneuver his body dismounted, 
including walking, running, and getting into and out of the prone position. He 
must be able to use personal gear available in the simulation, including compass, 
binoculars, global positioning system (GPS), watch, and night vision goggles 
(NVGs). The user must be able to use personal weapons, including rifle and 
grenade, to accurately engage targets. Other individual skills include recognition 
of civilians and enemy combatants, recognition of improvised explosive devices 
(lEDs), and casualty evacuation. In convoy operations. Marines must be able to 
operate vehicles and vehicle weapons proficiently. Specialized vehicle 
operations necessary for the convoy operation include towing a damaged 
vehicle. 

Typically, Marines will operate in a convoy scenario in teams of 
three; a vehicle commander riding in the passenger seat, a driver, and a gunner 
for the vehicle’s weapon system. The vehicle commander does not have to learn 
any specific vehicle interface skills like the driver or gunner, but he must be 
familiar with both. For this reason, the interface training was developed for 
teams of two, specialized for the driver and the gunner. Vehicle commanders 
could be trained using either station. 

The interface training scenario involved two parts: individual 
training and vehicle training. For the individual training, two Marines operated in 
the same environment that included a single personal weapons range. Users 
started in two separate lanes that mirrored each other with the range in the 
middle. Each user started play in front of his own vehicle. At this position, the 
user could learn individual movements. Both users moved to firing points on the 
personal weapons range and practiced M16A4 rifle and grenade target 
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engagement. Also, users practiced using their personal gear. After completing 
these exercises, users returned to their individual vehicles to start the vehicle 
training. 

Users started the vehicle training by driving high mobility multi¬ 
purpose wheeled vehicles (HMMWVs) along a designated course to a vehicle 
weapons range. There, they used the vehicle’s 0.50 caliber machine gun to 
practice engaging vehicle targets. After practicing with the vehicle’s weapon 
system, the users followed a designated course to a middle Eastern-style house 
where they could get out of the vehicle and examine each other, three civilians, 
and two enemy insurgents. Each user shot an insurgent and practiced putting 
the body in a vehicle. Users also viewed an lED and practiced disarming it. 
After personnel and lED recognition, the users got in a 7-ton truck, the medium 
tactical vehicle replacement (MTVR), and followed a designated course to a 
damaged Landrover. There, they practiced towing the vehicle. After completion 
of the vehicle towing, the users had five to ten minutes of free practice time. The 
entire familiarization course, shown in Figure 3, occupied a 3 by 4 kilometer area 
of the Cleghorn region of the Twentynine Palms terrain database available in 
VBS 2™. 
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Figure 3. Familiarization training area 
We designed an evaluation course in the familiarization scenario to 


determine the interface skill level achieved by the users (Figure 4). Users started 
in a fresh MTVR at a known command post location and used signs to follow a 
route on a mission to tow a damaged FIMMWV. Along the route, they 
encountered four stations briefed immediately prior to the exercise. Station 1 
included three enemy trucks, Station 2 included a four-man enemy fire team. 
Station 3 included a compound with two enemy combatants inside and a well- 
marked lED outside, and Station 4 included another four-man enemy fire team. 
A properly executed exercise would result in the three trucks destroyed, all ten 
personnel targets killed, one of the enemy from the compound loaded into the 
truck, the lED disarmed, and the damaged HMMWV in tow with the MTVR on the 


63 















Damaged Hi yifV WV 


Fire tean 2 


^jtnpotiml with 2«nemyand lED 


earn 


i Trainirig Areav, 


nearest road ready to return. All users had the following resources: a map of the 
area, a diagram of the familiarization training area, and a cheat sheet of interface 
commands. 


Figure 4. 


Team evaluation course 


In order to assess the degradation of interface skills over time, the 
project included a follow-up evaluation one week later. The exercise had users 
shoot an enemy insurgent, drive a vehicle to the body, and load the body into the 
vehicle. They destroyed an enemy truck with a grenade. Then the users 
assumed a prone position, shot a target less than 100 meters, and shot a target 
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farther than 100 meters. Users followed the instruction sheet included in 
Appendix I to keep track of each task accomplished. All participants used 
identical courses (in design, a prototype course was copied and pasted to make 
the others) about 200 meters in length in VBS 2™’s Samawah terrain database 
(Figure 5). All users had an instruction sheet that explicitly listed each step of the 
exercise. Half of the users had the cheat sheet of interface commands from the 
week before; the other half had no interface assistance. 



Figure 5. Individual evaluation course 

VBS 2™ offers a robust after action review (AAR) tool for debriefing 
training exercises. This tool allows the trainer to replay a recorded exercise 
using a viewing camera that can be positioned anywhere in the scene. The 
trainer can attach the camera to an object, such as a vehicle, or fly the camera 
through the scene to any point. The trainer can start, stop, fast forward, and 
rewind the AAR as required. In this way, a trainer can review any part of a 
tactical scenario multiple times from multiple positions. The AAR tool includes a 
timer enabling the recording of specific event durations. The project used this 
AAR tool for data collection in support of the evaluation. 

Participants completed a one-page pre-exercise survey that 
included demographic information and computer usage profile. They chose one 
of the twelve simulation computers available in an open classroom. After a brief 
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project introduction, the participants saw the slide presentation and completed 
the individual training portion of the exercise. They were then informed that they 
were operating as teams of two and started the vehicle familiarization training. 
After the personnel and lED recognition, the participants boarded the same 
vehicle and completed the rest of the training together. They completed the 
towing training and free practice as described above, proceeded to the command 
post, and boarded the evaluation MTVR. Participants had ten minutes to 
complete the evaluation exercise, recorded with the AAR tool. After the exercise 
finished, participants completed a one-page post-exercise questionnaire that 
focused on skills that were easy and difficult, memory of interface skills, and a 
seven-question Likert scaled evaluation of confidence in the basic interface skills. 
The responses to the basic skills involved a rating from 1 to 5, and we 
considered a rating of 3 as average for determining the success of the training. 
The two surveys used for the experiment are included in Appendix H. The 
complete exercise, including training and evaluation, lasted for a 50-minute class 
period. The follow-up evaluation occurred one week later, in the first five minutes 
of the class period. 

3. Results 

First and foremost, a pilot study serves to identify unforeseen problems in 
the project methodology. This pilot study accomplished this goal with the 
following observations. Because of a technical software issue, disarming an lED 
crashed the AAR tool. For this reason, all participant activity recorded after lED 
disarming was lost to analysis. The AAR tool shows shots fired on a time bar, so 
one could determine whether the team fired after the lED crashed the AAR tool 
even though the lED disarming prohibited viewing this portion of the scenario. 
Lack of clarity in the instructions resulted in participants driving their HMMWVs 
instead of the MTVR to the Landrover for the towing training. The HMMWVs 
could not tow, so the participants did not train that skill, but had a verbal 
explanation of the towing procedure. Participants tended to assault the insurgent 
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fire teams. Although the enemy objects were set to “never fire,” they fired to 
defend themselves when in imminent danger, resulting in unexpected close- 
range small arms battles. Finally, in the second session evaluation, one of the 
participants shot another right at the beginning of the scenario, so that the victim 
participant was removed from all data collection. These unexpected problems 
confounded the data collected from the evaluation exercise. 

a. Team Evaluation 

Three of the six teams completed the full evaluation with a mean 
time of 7:11 (standard deviation 1:32). Four of the six teams completed the 
evaluation through the lED station with a mean time of 6:54 (standard deviation 
1:30). All six teams engaged and destroyed the three trucks at the first station. 
Four of the six teams assaulted Fire Team 1 by driving right up to the enemy. As 
previously described, the artificial intelligence allowed the enemy to defend itself 
when in imminent danger, resulting in close-quarters battles with Fire Team 1. 
For this reason, one team lost both members at the Fire Team 1 station and 
could not continue. Two teams continued the scenario with one member dead; in 
both cases, the single team member completed the scenario alone. 

All six teams eventually followed the correct path as designated by 
the signs. However, four of the six teams showed signs of disorientation at some 
point during the exercise. Two of the teams strayed so far from the path that 
they clearly relied on their maps to navigate back. 

As mentioned, three teams did not complete the course. One team 
did not complete the course because Fire Team 1 killed both members. For the 
team that completed the course through the lED station, but did not complete the 
course entirely, one of the team members faced a unique problem. The team 
member was in the prone position and remained that way. The member could 
move forward by low crawling, but could not stand. This behavior indicated that 
the team member had become injured, although it was unclear how this had 
happened. The member’s inability to stand made disarming the lED impossible. 
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The third team chose to disarm the lED at the compound before engaging the 
enemy inside and evacuating the casualty. For this reason, the AAR tool did not 
capture any further action at the compound. However, we could determine that 
the team fired on Fire Team 2 later in the scenario. 

b. Survey Results 

The post-exercise survey provided some insight into how 
participants felt about their training. Participants answered two knowledge 
questions on the post-exercise survey. Without the aid of the simulation or the 
cheat sheet, all participants correctly knew how to move their body forward. 
However, only five of the twelve participants correctly explained how to tow a 
vehicle. Of these five correct respondents, four were drivers. 

Participants responded to two subjective questions about the skill 
or skills they found most difficult and the skill or skills they found easiest. Nine of 
the twelve participants found shooting easiest, and two found body movements 
easiest. Answers to the question about the most difficult task differed 
considerably and included the following: towing, disarming an lED, driving 
backward, navigation, confirming targets, unslinging a weapon, and recovering a 
casualty. A group interview question immediately following the exercise revealed 
that reported driving problems were associated with towing, because the driver 
must maneuver around the damaged vehicle and back up to it. The driver has 
no mirrors or other means to see behind him, so another individual in the 
simulation must guide him. 

Participants responded to a question about their comfort level with 
each of the interface skills on a five-point Likert scale, with large numbers 
reflecting the highest degree of confidence in that skill. Figure 6 shows the 
results for the following six skills: basic body movement, vehicle maneuvering, 
shooting individual weapons, shooting vehicle heavy weapons, manipulating a 
casualty, disarming an lED, and towing a vehicle. Participants felt least confident 
with the casualty evacuation and towing tasks. Review of the AAR tool 
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recordings confirms that these tasks were more difficult than shooting and 
moving. Confidence ratings exceeded an average of 3.0 for all tasks except 
towing. 


ConfidenceAverages 


6 


5 


4 


Average 


2 


1 


0- 

Body 

Veh 

Shoot 

MG 

Cas 

Tow 

lED 

ConfidenceAverages 

4.08 

3.92 

4.75 

4.58 

3.33 

2.82 

3.83 



Figure 6. Subjective ratings of confidence in interface skills 

c. Follow-on Individual Evaluation 

The follow-on evaluation conducted one week later demonstrated 
considerable degradation of previously taught interface skills. Only eleven 
participants conducted the exercise, because one participant shot another 
immediately at the beginning of the exercise, eliminating the victim from further 
play. Only two of the eleven remaining participants completed the exercise 
correctly. An additional participant completed all of the events except standing 
and recording his watch time. He accomplished this in less time than the two 
who completed the exercise. This participant was a foreign officer and might not 
have understood the instructions fully to complete the last two events. Ten of the 
eleven participants shot the first target (the eleventh successfully shot a target, 
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but it was another participant). All eleven participants successfully interacted 
with the HMMWV. Seven of the eleven participants successfully loaded the 
casualty into the HMMWV. Only four participants threw a grenade at the truck 
target, and those four participants also successfully engaged the short and far 
targets. Five of the eleven participants successfully used their simulation watch 
to record a time. 

During the follow-on individual evaluation, half of the participants 
had the cheat sheet from the previous week’s training, and half did not. Of the 
four participants who successfully completed the exercise through shooting the 
far target, two had cheat sheets and two did not. Both the participant who shot 
another participant and his victim had cheat sheets. The four participants who 
shot the far target had times of 1:54, 2:23, 2:40, and 5:08. 

The follow-on individual evaluation employed robust measures to 
guide participants through the exercise. Participants had a sheet with all 
instructions explicitly listed. The course was simple; no other objects cluttered 
the terrain except the targets for the scenario. Cones provided a maneuver box 
for each participant. Despite the clear verbal and written instructions and visual 
cues in the simulation, one participant still managed to get disoriented and 
confused. Not only did this participant kill another, he drove across another 
participant’s area and loaded the victim in his HMMWV. 

d. Practical Tips for Foiiow-on Work 

In addition to determining whether the proposed interface training 
sufficed, the pilot study provided an opportunity to investigate practical aspects of 
conducting research with VBS 2™. The Marine Corps intends DVTE, including 
VBS 2™, to be portable. The pilot study site was set up for the single class 
period of the exercise only. We set up the system twice in this project: once for 
the training session and once for the follow-up evaluation. We conducted an 
equipment rehearsal prior to the first session of the pilot study, which provided 
several lessons learned. Based on the equipment rehearsal, we diagrammed the 
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pilot study site with the intent to set up all twelve computers, network them, and 
run the scenario in a 50-minute period. Our setup script and diagram that 
resulted from the equipment rehearsal are included in Appendix G. We found 
that we had all hardware set up and networked within 25 minutes for both 
sessions. Timing for starting scenarios varies depending on the terrain database 
that is loaded and the networking arrangement. For example, the Twentynine 
Palms database requires much more processing power than any of the other 
installed databases. Because the first session involved six separate 
environments (one for each pair of participants) and the Twentynine Palms 
database, we took 15 minutes to get all computers running. With a single 
environment and the Samawah database for the second scenario, we only 
required five minutes to start the scenario. In both sessions, with the help of the 
twelve participants, all twelve computers were shut down and stowed within ten 
minutes. This timing information is critical for future research efforts that will 
involve setting up a group of computers and stowing them after the experiment. 

During the team evaluation exercise, one participant experienced 
mild motion sickness and had to stop participation in the exercise for a short 
time. The participant filled the role of driver. He reported that he had 
experienced motion sickness in simulators before, and he had used simulations 
in professional training. 

4. Discussion 

This pilot study sought to develop a training program that could prepare 
novice users to properly and efficiently use the VBS 2™ interface to conduct 
tactical convoy operations within 30 minutes. For our survey success criterion, 
the training resulted in average or above average comfort levels for all skills 
except towing. Because of the technical problems in teaching the towing task, 
we eliminated towing from the criteria and concluded that the training met the 
success criterion. While the pilot study highlighted some concerns that must be 
addressed for future work, the results of the project confirmed that the scenario 
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and training construct sufficed to familiarize users with the interface. The 30- 
minute training period can be used for timing planning for future research. 

The 30-minute training period most successfully taught the individual skills 
of moving and shooting, although participants had difficulty cycling between 
individual weapons in the individual evaluation. The more complex skills of 
casualty evacuation, towing, and disarming an lED caused the most problems. 
However, when the participants worked in teams of two, they generally figured 
out how to deal with these tasks as a team. As a team, participants experienced 
more trouble with navigation and surviving a threatened enemy in close quarters. 

Observation of the trainees throughout the group and individual 
evaluations indicated that participants had sufficient training but needed more 
practice. The explanation and demonstration sufficed to give participants a basic 
understanding of how to use the interface. The cheat sheet provided a sufficient 
resource to fill in gaps in knowledge for those tasks not exercised regularly. 
Participants simply needed to practice more to learn the interface and the cheat 
sheet. Thus, the success of the interface training must be measured relative to 
what will succeed it. Participants do not complete the interface training ready to 
be tested in a larger exercise using VBS 2™ as an extension of their warfighting 
skills. However, the interface training prepares them to start working on more 
complex tactical tasks that would inherently provide the interface practice needed 
for larger exercises. Estimating the amount of practice necessary to use the 
interface proficiently is beyond the scope of this project, but this study indicated 
that the Aussie research team cited in this thesis’s background provides a correct 
estimation of two hours or less for the average user. 

The errant participant in the follow-on individual evaluation highlighted the 
issue of fratricide. Despite the care taken to expose participants to the different 
types of avatars in VBS 2™, one participant did not correctly distinguish friendly 
targets at close range. This incident suggests further emphasis on the avatar 
recognition portion of the training. 
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For future repetitions of this experiment, some means of minimizing the 
confusion experienced by the errant participant must be devised. Based on this 
project, two potential solutions suggested themselves. First, the training scenario 
used a waypoint at the beginning to guide users to the shooting range. The 
waypoint provides an arrow in the user’s screen, indicating the direction he 
should go. The waypoint also provides a distance to travel. Waypoints should 
be used to help the participant find his way. Second, the AAR function could 
potentially help participants. The researcher could conduct the follow-on 
individual exercise himself, using the AAR to record an example of a correct 
execution. The researcher could then play this AAR prior to the follow-on 
individual evaluation to provide the user a visual demonstration of how the 
exercise should be done. From that point, the participant’s only task would be to 
manipulate the interface to repeat what he had just seen. This visual display 
may help the user avoid the confusion of the scenario, thereby focusing on the 
interface tasks. 

The unintended problem in the towing training, in which participants 
moved to the Landrover in the wrong vehicles, provided a valuable comparison 
study. The towing operation involves steps roughly comparable to loading a 
casualty into a vehicle. Of all the interface skills, participants were most 
uncomfortable with towing. Because most participants achieved proficiency in 
skills of similar complexity, the value of hands-on practice during the training 
became clear. Flaving participants follow along in their own simulation 
environments during the brief appeared to produce better results than a verbal 
explanation of the procedures. 

The empirical data from the study did not demonstrate that participants 
depended on the cheat sheet. Nevertheless, participants conducted an exercise 
for fun after the follow-on individual evaluation. All of the participants without the 
cheat sheet were grateful to get it back for the next exercise. Participants were 
observed consulting the cheat sheet during the exercise. 
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A pilot study primarily provides lessons learned for future work. This study 
highlighted some issues that could endanger a future research effort if not 
corrected. Specific issues identified in this pilot study include: 

• Ensure that lED disarmament does not interfere with AAR 
recording. 

• Brief participants to remain on the road, so they do not provoke the 
enemy targets to engage them in close-quarters battle. 
Alternatively, make the enemy targets impotent by reducing their 
ammunition levels to zero. 

• In the towing training, clarify vehicle mounting so that participants 
use the MTVR. 

• Screen for previous simulator sickness, and prepare training units 
for the fact that some participants may not be able to handle the 
simulator. 

• Brief rules of engagement for the evaluation exercise (that is, 
engage all targets upon identification). 

• Emphasize the key control to raise and lower the rifle. 

• On the survey, the computer use question implies the individual’s 
personally owned computer. Divide the question into personally 
owned computer use, and government and other computer use, to 
encompass all of a participant’s computer time. 

• On the survey, make a block for the computer number clearly at the 
top of the page for clarity. 

• VBS 2™ offers two modes of operation: user and administrator. 
Connect participants as users instead of administrators. One 
participant was able to learn how to use administrator rights to 
interfere with the conduct of the exercise. Connecting the 
participants as users also allows the assignment of unique names, 
which would make data collection much simpler. 

Some notable features of the pilot study that worked well deserve 
consideration in future work: 

• The general construct of the presentation combined with hands-on 
practice worked well for all participants. The familiarity training 
scenario was constructed well to support the training. The cheat 
sheet worked well as written. 

• The equipment rehearsal provided valuable lessons learned. Had 
this not been done, a number of potential pitfalls could have 
jeopardized the pilot study before it even started. 
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• Two resources supported the researchers setting up the pilot study 
setting. A script of all the tasks that needed to be accomplished 
provided explicit instructions for setup. A diagram of the computers 
and wiring connections accompanied the script. The assistant 
never consulted the text of the script, but relied heavily on the 
diagram. Future VBS 2™ projects will involve various assistants. 
More effort should be put into wiring schematics, and the script 
should not be provided at all. 

• For the researcher conducting the study, a script was absolutely 
necessary. The interface training, group evaluation, and follow-on 
evaluation, were scripted in detail and well rehearsed. The script 
and rehearsal were valuable tools to minimize critical errors in the 
conduct of the study. 

In summary, use of VBS 2™ as a tactical training tool depends heavily on 
interface training. Without adequate interface training and experience. Marines 
will not be able to use the simulation as an extension of their warfighting skills, 
resulting in unintended problems. While a 30-minute training session does not 
make a novice into a proficient VBS 2™ user, the training suffices to get the 
person started so that the individual only needs to practice to achieve interface 
proficiency. Interface training works well if trainees can work hands-on in the 
simulation during presentation. A brief cheat sheet does not make a user 
proficient by itself, but provides support when needed. This study provided 
several practical insights for future work. Key improvements involved technical 
issues with the AAR tool and connecting computers in user mode. Key 
successes included the value of preparation and rehearsal and the importance of 
an exercise script. Finally, the study provided hints into how confused and 
disoriented a trainee may become in even the simplest of VBS 2™ simulations— 
a consideration that will certainly impact future work. 

B. SCENARIO DEVELOPMENT PILOT STUDY 


1. Introduction 

This section summarizes a second pilot study conducted as part of a 
project to examine the training effectiveness of VBS 2™ to train small unit tactics 
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in a personal computer-based environment. This project involved the testing of a 
specific small unit tactical convoy scenario. In evaluating the scenario, we 
sought to determine whether the training improved the participants’ appreciation 
of serious games as a training tool. Furthermore, we investigated whether 
participants felt that their small unit tactical convoy knowledge and skills 
improved as a result of the training. These questions served as a first step 
toward determining whether VBS 2™ can provide effective tactical training for 
military personnel. 


a. The Training Exercise that Started it Ait 

At the Marine Corps Air Ground Combat Center (MCAGCC) in 
Twentynine Palms, the Tactical Training Exercise Control Group (TTECG) 
administers a series of training scenarios to operational units. The TTECG has 
developed these scenarios over time against the backbone of the training and 
readiness system. As a starting point for our scenario development, we 
observed TTECG operations. TTECG supports a series of exercises packaged 
under the name Mojave Viper. Marine Corps fleet units, both active and reserve, 
rotate through Mojave Viper training at the rate of twelve units per year. TTECG 
exercises are rigidly controlled and evaluated in the effort to provide all units the 
same training experience and a common evaluation baseline. The Motorized 
Operations Course (MOC) provides a tactical scenario with four individual 
stations to test the standard operating procedures, immediate action drills, and 
small-unit tactics of a platoon-sized convoy (usually between six and eight 
vehicles). 

The first station tests the unit’s ability to recognize an lED, secure 
the area, report the appropriate information to higher headquarters, and link up 
with forces who will handle the device. The station consists of a vehicle hulk 
representing a vehicle with an lED inside. The second station requires a unit to 
react to an exploding lED and a far ambush. The exercise controller causes an 
explosion (one-pound stick of TNT) and informs the unit that the simulated bomb 
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disabled the lead vehicle. The controller then activates ten personnel targets at 
long range (350 meters). The participating unit can choose to suppress the 
targets and maneuver the unit through or suppress the targets and assault. The 
third station consists of a destroyed enemy battle position with a near ambush 
consisting of trenches with bunkers and personnel targets. Again, the unit may 
choose to suppress and maneuver through or assault the targets. Regardless of 
the drill employed, the controller will assess one or more casualties for the unit to 
handle. If supporting aviation assets are available, the controller will paint enemy 
units to the north. Station 4 allows the unit to exercise control of tactical aircraft 
to destroy two vehicles. 

The MOC provides the opportunity for the exercise unit to practice 
several distinct skills. The unit must execute “fire and maneuver,” the set of skills 
used to direct and control multiple vehicles to destroy targets while maintaining 
unit security. All vehicles must practice fire discipline so that all targets get 
destroyed without expending all ammunition on one target and letting others go 
free. Internal communications must be correct and effective. The unit must 
exercise control over intelligence and combat air support assets in a tactically 
meaningful way. The unit must handle battlefield realities such as casualties and 
vehicle breakdown. The unit must coordinate its actions with higher 
headquarters, reporting all significant activity as appropriate. The exercise tests 
the unit’s understanding and practical application of convoy tactics. The exercise 
unit demonstrates the ability to conduct the various immediate action drills 
required for tactical success. Most importantly, the MOC provides an opportunity 
to scrutinize a unit’s standard operating procedures to determine whether they 
are really standard, and whether they are truly understood. 

In Chapter II, we discussed NAVMC 3500.87, the Infantry Training 
and Readiness (T & R) Manual, that serves as the basis of all infantry training. 
The publication codes and lists individual tasks as Mission Performance 
Standards (MPSs). Using the events of the MOC as a model, we developed a 
list of individual tasks to train. The T & R Manual summarizes convoy operations 
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into a single platoon collective task INF-MOBL-5150 “Conduct convoy 
operations.” We broke this task down into subtasks that participants could easily 
evaluate. The T & R Manual supports each of these tasks, although we 
separated some subtasks for the sake of evaluation convenience. Table 1 
shows our final list of subtasks and traces their origins in the T & R Manual. 


Evaluated Event in Experiment 

T & R Manual Source 

React to unexploded Improvised 
Explosive Device 

INF-MOBL-3150 React to an 
unexploded Improvised Explosive 

Device (lED) 

React to an Improvised Explosive 

Device detonation 

INF-MOBL-3151 React to an 

Improvised Explosive Device (lED) 
detonation 

Take immediate action against a 
blocked ambush 

INF-MOBL-5155 Take immediate 
action against blocked ambush during 
motor march 

Take immediate action against an 
unblocked ambush 

INF-MOBL-5156 Take immediate 
action against unblocked ambush 
during a convoy 

Cordon and 360 degree security 

Component event 9 of INF-MOBL-5150 
Conduct convoy operations 

Employ vehicle machine guns / 
weapons 

INF-WPNS-5308 Conduct heavy 
machinegun offensive operations 

Mounted fire and maneuver 

INF-WPNS-5308 Conduct heavy 
machinegun offensive operations 

Shift fires / cease fires 

INF-WPNS-5308 Conduct heavy 
machinegun offensive operations 

Vehicle recovery / bump plan 

Component events 11 through 15 of 
INF-MOBL-5150 Conduct convoy 
operations 

Casualty evacuation 

INF-MED-5430 Process casualties 

Communication with higher 
headquarters 

Component event 7 of INF-MOBL-5150 
Conduct convoy operations 

Communication between vehicles in 
convoy 

Component event 7 of INF-MOBL-5150 
Conduct convoy operations 

Communication between personnel in 
vehicle 

Component event 7 of INF-MOBL-5150 
Conduct convoy operations 


Table 1. Evaluated tasks in convoy training experiments 
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We did not use the MOC itself in this study, but one must 
understand the idea of the exercise as background to this work. This exercise 
provided the inspiration for the scenarios we developed. Also, the MOC inspired 
the live exercise used as an evaluation mechanism in a follow-on experiment. 
Our analysis of this exercise included personal observation and detailed analysis 
of the grading sheet used to evaluate a platoon’s performance. The MOC is 
developed on solid Marine Corps doctrine and tactics, techniques, and 
procedures. We based the entire body of work discussed in this thesis on a real 
operational exercise conducted by each unit in the Marine Corps at some point in 
the training cycle. 

b. Background Literature Supports Survey Techniques 

The ultimate test of the effectiveness of a serious game as a 
training tool is improved performance in the real world. However, this approach 
is not the most practical initial step in the laboratory environment. For this 
reason, the bulk of the study of serious game effectiveness has involved user 
self-assessment. The Aussie studies of VBS 2™ discussed in Chapter II provide 
some of the most valuable published insight into the utility of VBS 2™ as a 
training tool. Table 2 summarizes all of the previously referenced Aussie VESL 
studies. Table 2 highlights a trend in measurement technique across the 
spectrum of the research work: that is, a consistent use of participant 
questionnaires to determine how well the simulation worked. 
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Year 

Project 

Independent 

Variable 

Dependent 

Variable 

Measurement 

Technique 

Title: 

1^^-person tactical shooters: COTS games with military training potential? 

2002 

Determine 
whether VBS 2™ 
had military 
training potential 

None 

None 

13 question survey 
of 12 SME’s 

Title: 

Constructing the virtual section 



2004 

Examine decision 
making 
techniques 
employed in 
game play 

Amount of 
verbal 

communication 

Use of roles 

Awareness of 
team 

disposition 

Performance 

Points scored in 
game; participant 
questionnaire 

Title: 

Factors in team performance in a virtual squad environment 

2004 

Determine the 
ideal squad size 

Size of section 

Performance 

Loss Exchange 

Ratio; participant 
questionnaire 

Title: 

The use of games to investigate tactical decision-making 

2005 

Examine decision 
making models 
employed in 
game play 

Decision 

making 

approach 

Performance 
in training 
scenario 

Correlation: 
decision making by 
questionnaire, 
performance by 
number of win/lose 
matches won 

Title: 

Challenging the super soldier syndrome in person 

simulations 

2005 

Examine the 
extent of Super 
Soldier 

Syndrome in 

VBS 2™ 

Use of 
suppression; 

Use of 
accuracy / 
dispersion 

Change in 
game tactics 
(tempo of 
movement, 
use of cover, 
firing 

procedure, 
personal 
tactics, team 
tactics) 

Points scored in 
game; participant 
questionnaire 
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Title: Proficient soldier to skilled Gamer: Training for COTS success 

2005 

Determine how 
long it takes a 
user to be able to 
use VBS 2™ as 
an extension of 
warfighting skills 

Time to learn 
interface 

Performance 

Pass a “Test of 
Objectives” 

Questionnaire 
asking when 
participants felt 
comfortable 

Title: Heart-rate and immersion in a first-person simulation 

2006 

Determine 
physiological 
response and 
sense of 
presence in VBS 

^TM 

Activity (rest, 

walking, 

playing 

different 

scenarios) 

Physiological 
response and 
sense of 
presence 

Fleart rate with a 
monitor and 7 
question survey 

Title: After action review of simulation results: impact of presentation modality 

2006 

Examine different 
visualization 
modalities to 
enable a 
commander to 
effectively use 

VBS 2™ for AAR 

AAR 

presentation 

modality 

Utility of 
modality 

Questionnaire 


Table 2. Summary of performance measurement techniques 

used in Aussie studies 


In Chapter II, we discussed two Naval Postgraduate School 
projects involving serious games. In 2005, U.S. Army Majors J. Nolan and J. 
Jones used Delta Force: Black Hawk Down-Team Sabre™ as a small unit tactics 
training platform at the Infantry Officer Basic Course in Fort Benning, GA. The 
two researchers compared survey data from a control group who did not use 
simulation and a test group who used the COTS game to demonstrate that 
serious games could be used for training (Nolan & Jones, 2005). In 2007, U.S. 
Marine Corps Major Neil Fitzpatrick and Turkish Army Captain Umit Ayvaz used 
a similar survey mechanism to determine the effectiveness of a decision-making 
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trainer called Close Combat Marine (Fitzpatrick III & Ayvaz, 2007). In both 
research efforts, user feedback and self-assessment provided the sole basis for 
effectiveness analysis. 

In summary, relevant literature indicates that the first sensible step 
in determining the effectiveness of a simulation trainer is to test user appreciation 
of the serious game. Previous work demonstrates how a researcher can 
leverage survey techniques to gain insight into the effectiveness of the training 
tool. We proposed to conduct a small unit tactical convoy exercise with VBS 2™ 
and see how participants’ perceptions of the serious game and their own tactical 
skill changed. 


c. Research Interests 

As a pilot study supporting a larger project, this project served two 
main purposes. First, as the background literature indicates, progress in future 
VBS 2™ effectiveness evaluation projects requires investigation into user 
perceptions of the serious game. The larger supported project focused on small 
unit convoy tactics. Thus, this project started with the development of tactical 
scenarios and tested them with two user groups. We conducted the scenario 
testing to gain insight into the following two null hypotheses: 

• The use of VBS 2™ does not change the appreciation of 
serious games as a training tool, as measured by the Likert 
scaled responses of participants. 

• Participants feel that the use of VBS 2™ does not change 
their small unit convoy knowledge and skills, as measured 
by Likert scaled responses. 

Second, for planned future projects, groups and teams will use VBS 
2™ as a common environment to interact toward some unit goal. As in the case 
of the interface pilot study discussed in the previous section, a lone researcher in 
a laboratory can only troubleshoot a limited number of issues in support of such a 
project. Common sense dictates that many unpredicted problems and quirks will 
arise from group use that the lone laboratory developer can never find by himself. 
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In this regard, this pilot study served as the first effort in the larger project of 
having multiple people work together in a common environment. 

The TTECG MOC exercise involves Marines patrolling a 
designated course in six to eight vehicles armed with heavy weapons. The 
course trains the following core skills: react to an unexploded improvised 
explosive device (lED), react to an lED detonation, take immediate action against 
a blocked ambush during a convoy, take immediate action against an unblocked 
ambush during a convoy, evacuate a damaged vehicle, and evacuate a casualty. 
We established these skill sets as the core objectives of our simulation training. 
From these skills, we derived a series of three progressively more difficult 
scenarios. The pilot study trained participants with these scenarios, comparing 
their attitude toward serious games and their self-assessed tactical skill before 
and after the training. 

2. Method 

a. Participants 

We conducted the pilot study in two distinct parts. The first part. 
Experiment 2, consisted of testing a single convoy scenario in a school 
environment at the Naval Postgraduate School in Monterey, CA. The second 
part. Experiment 3, tested all three scenarios with operational Marines from a 
training command in Camp Pendleton, CA. Experiment 2 served to work out the 
technical issues in running the scenario, such as testing radio operation, 
ensuring participants matched with the right avatars in the game, and checking 
networking ideas for feasibility. Experiment 3 gave Marines with convoy 
experience the opportunity to comment on the tactical validity of the scenarios 
and VBS 2™ itself. We evaluated both groups for change in attitude toward the 
simulation and change in self-assessed tactical proficiency. 

Experiment 2 was broken into four 50-minute sessions conducted 
weekly for one month. Twenty Naval Postgraduate School Modeling, Virtual 
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Environment, and Simulation (MOVES) graduate students, ranging in age from 
27 to 41 years, participated in the pilot study. All participants volunteered as part 
of a seminar program within the curriculum. All the participants were male 
military officers with service times ranging from 4 to 24 years. Three foreign 
officers participated, and U.S. officers represented all four of the armed services. 
The first session provided training on the use of the VBS 2™ interface focusing 
on the following tasks deemed applicable to the convoy scenarios: individual 
body movements, manipulation of personal gear, use of personal weapons, 
vehicle interaction and operation, use of vehicle weapons, manipulating a 
casualty, towing a vehicle, disarming an lED, and recognition of friendly, civilian, 
and enemy avatars. The second session provided a short 20-minute overview of 
convoy tactics. Participants then had a 15-minute practice period to practice 
mounting vehicles in the assembly area, forming into march formation, tactical 
movement in a benign environment, and control of fires on a target array. During 
the third and fourth sessions, the participants conducted a tactical convoy 
scenario. 

We conducted Experiment 3 in a full day of training on site at 
Weapons and Field Training Battalion (WFTBN) San Diego in Camp Pendleton, 
California. Our training schedule for the day is included in Appendix K. Twenty- 
four Marines, ranging in age from 19 to 29 years, participated in the study. Unit 
leadership randomly selected eight Marines from each of the three companies for 
participation in the training. The battalion conducts marksmanship and field 
training for recruits, so approximately two-thirds of the participants were infantry, 
and the other third were support personnel including communicators, armorers, 
and supply and administration personnel. The training group consisted of one 
Private (El), one Private First Class (E2), three Lance Corporals (E3), nine 
Corporals (E4), nine Sergeants (E5), and one Staff Sergeant (E6). The rank 
structure provided us the ability to structure the personnel like a typical Marine 
platoon, although the Marines had not operated together before. We chose 
WFTBN because of the combat experience of the Marines in the unit. WFTBN is 
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a garrison training environment that serves as an opportunity for Marines to get a 
break from heavy operational deployment cycles in the fleet forces. Only five of 
the Marines had never been to combat. Eighteen Marines had participated in 
convoy operations in combat, and twelve Marines reported more than 100 
personal convoy operations in combat. As a group, the WFTBN Marines were 
Subject Matter Experts in the tactical skills included in this study. The second 
study included the same exercise progression as the first. Marines started with 
the basic interface training and then moved on to the progressively more difficult 
tactical convoy scenarios. 

b. Apparatus 

The project used twenty Dell Precision M6300 laptops from a suite 
of the Marine Corps Deployable Virtual Training Environment (DVTE) package. 
Computers were networked using D-LINK DGS-2208 8 port switches. Peripheral 
equipment, including mice, cables, and switches came from the standard DVTE 
package. 

We used a standard classroom for Experiment 2 with the MOVES 
students. For the familiarization training and evaluation, we networked the 
computers in pairs with four computers per switch. Paired users faced each 
other at a classroom table so that they could not see each other’s screens. For 
the convoy exercises, we connected the computers on a single network with six 
computers per switch. Participants sat in classroom seating in a theater-style 
classroom. The three members of each truck crew sat in a row, but were 
dispersed so that they could not see each other’s computer screens. The 
diagram of the room setup is included in Appendix J. Participants communicated 
using the Joint Virtual Tactical Radio (JVTR), a voiceover IP system. JVTR 
allows the use of two communication nets, so vehicle commanders 
communicated on one net, while the other net was used as a vehicle intercom for 
the truck crew to communicate. Figure 7 shows the setup of the room for the 
convoy exercises. 
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Figure 7. Tactical convoy training at the Naval Postgraduate School 


For Experiment 3 conducted at WFTBN, we used a long, narrow 
classroom in the unit’s academic building. The facility featured overhead 
projection support and had more than enough room for the twenty-four 
participants to spread out. We used about half of the room’s individual desks 
and interspersed them around the room to seat participants in three-person 
groups with maximum space between groups. We used a setup similar to the 
MOVES study with computers networked at five to eight per switch, depending 
on the physical locations of the machines. Generally, participants could not 
easily see each other’s screens, although the setup did not prohibit the 
determined Marine from doing so. We used the same system for JVTR 
communications that we used in the MOVES study, with two radio nets 
supporting vehicle and convoy communications, respectively. Figure 8 shows 
the WFTBN training site. 
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Figure 8. Tactical convoy training at WFTBN, San Diego 


c. Procedure 

We used a similar experiment procedure for both the MOVES and 
WFTBN experiments. Before the tactical scenario training, participants 
completed a two-page questionnaire, included in Appendix L. The questionnaire 
included demographic and computer usage information. This questionnaire also 
included a set of questions intended to determine the participant’s attitude toward 
personal computer-based simulation training. Another set of questions 
determined the participants’ perception of tactical skill. Responses to both sets 
of questions were recorded on a 5-point Likert scale. After the tactical scenario 
training, participants completed a three-page post-exercise questionnaire, 
included in Appendix N, that featured subjective questions about the 
effectiveness of the training. The simulation attitude questions and tactical 
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proficiency questions from the pre-exercise questionnaire were repeated in the 
second questionnaire. Comparison of these Likert scaled responses served as 
the basis of analysis for this project. 

We wanted to evaluate an academic test for later use, and we 
administered the test to the WFTBN group. We developed two similar tests with 
eighteen multiple choice questions each. The questions related to the general 
tactical areas of reaction to ambush, reaction to lED, casualty evacuation, and 
vehicle recovery. Each of the questions had four choices for an answer with one 
best answer. We developed the tests from the training materials administered for 
convoy classes at TTECG in Twentynine Palms and The Basic School in 
Quanico, VA, that trains all Marine Corps junior officers. We designed the tests 
to be similar but different with paired questions. For example, a reaction to 
ambush question on one test would have a similar question on the other test. 
The Marines took one of the tests before training and one after training. We 
randomized the test taken first, with half taking one of the tests first and the other 
half taking the other test first. We included both tests in Appendices P and Q. 

For all tactical scenarios, participants were grouped in teams of 
three: a vehicle commander riding in the passenger seat, a driver, and a gunner 
for the vehicle’s weapon system. Seven vehicles participated in the MOVES 
exercise, and eight vehicles participated in the WFTBN exercise. The vehicles 
consisted of a mix of five HMMWVs and two MTVRs armed with an even mix of 
0.50 caliber machine guns and MK 19 40 mm grenade launchers. Each tactical 
scenario consisted of a series of stations, including reaction to an inert lED, 
vehicle recovery, reaction to an unblocked ambush, casualty evacuation, and 
reaction to a blocked ambush. The scenarios used the game’s Twentynine 
Palms database and involved operation in the western Quackenbush area of the 
terrain. The scenarios varied from roughly 20 kilometers in length to a scenario 
involving more than 40 kilometers of the western side of the Twentynine Palms 
training area. 
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For analysis, we used statistical group comparison techniques 
including the paired t-test and the Wilcoxon rank-sum method. For all statistical 
tests, we used an alpha level of 0.05. 

3. Results 

First and foremost, this pilot study served to identify unforeseen problems 
in the project methodology. This pilot study accomplished this goal with the 
following observations: 

• Some users failed to take training seriously (negligent discharges / 
fratricide). This degraded the experience for the whole group, not 
just the individual. This problem was much more pronounced with 
the WFTBN group. 

• Communication was a burden as VBS 2™ has no internal 
communication tool. 

• Computer glitches hindered the commencement of training. 

• 12 computers = 1 man-hour setup -i- 1 man-hour startup -i- 1 man¬ 
hour breakdown; It takes a working knowledge to set up the system 
and run the program correctly 

While these unexpected problems confounded the data collected from the 
evaluation exercise, the survey results provided valuable feedback. 

a. Survey Results 

The post-exercise survey implied that VBS 2™ had little training 
value as far as both MOVES students and WFTBN Marines were concerned. 
MOVES students, having little to no convoy experience, suggested that learning 
immediate action drills was the only value provided by the game. The Marines, 
when tested, suggested the same thing. While the sample was relatively small 
and the Marines used for the exercise were part of a training battalion, over half 
of the Marines used were from the infantry occupational specialty. Also, most of 
the Marines had average to extensive convoy experience and could be 
considered “Subject Matter Experts” in convoy standard operating procedure and 
immediate action drills. When the entire sample of post-exercise results was 
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evaluated, there appeared to be no significance that would lead to rejecting 
either hypothesis, implying that there was no training value in using VBS 2™. 
However, when the data of Marines with little to no convoy experience was 
separated from the whole to more closely mirror the sample from the NPS trials, 
we found significance in some of the immediate action drills just as we did from 
the MOVES students. 

b. Academic Knowledge Test Results 

Only the WFTBN study involved the administration of the academic 
knowledge test. In this endeavor, we wanted to ascertain whether the test made 
tactical sense to our pool of Subject Matter Experts. We did not give 
consideration to the results of the academic knowledge test as an indicator of the 
potential effectiveness of the simulation training in this study because we did not 
design the study to improve academic knowledge of convoy operations. Rather, 
the study served to determine whether the scenarios we had developed met our 
technical and tactical expectations. With this in mind, our analysis of the 
academic knowledge test results did not focus on change in performance. 
Instead, we concerned ourselves with the fairness of the test, and whether the 
administration of the test produced data that could be analyzed in later studies. 

We recorded the results of the test in a spreadsheet. If Marines 
answered questions incorrectly, we recorded the incorrect response in the 
corresponding spreadsheet cell. With this data, we not only examined how many 
Marines missed each question, but whether the questions were answered 
incorrectly with consistent responses. In this way, we could generally group 
questions into three categories: questions that were answered correctly, 
questions in which Marines could not agree on an answer (incorrect and 
inconsistent answers), and questions in which Marines as a group disagreed with 
our answer (incorrect, but consistent answers). The last two categories of 
questions differed. Incorrect and inconsistent answers indicated that Marines did 
not understand the material addressed by the question. Incorrect and consistent 
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answers indicated that the Marines as a group had an understanding of the 
question but looked at the issue differently than we intended. Most likely, this 
final problem suggested a problem with the wording of the question. 


Tables 3 and 4 summarize the data from the knowledge tests. We 
started by highlighting questions that we considered a problem, and we used a 
criterion of less than 60% of the Marines answering the question correctly. Using 
this idea, we noted that Test 1 had six problem questions while Test 2 had eight. 
Next, we analyzed the responses to look for problem questions that may have 
resulted from unclear wording. We noted that Question 4 from Test 1 and 
Questions 3, 13, and 16 from Test 2 particularly fit in this category. These four 
questions were noteworthy because they all related to organization of the convoy 
and division of labor within the convoy organization. 


Question 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 


Number respondents for each answer Yellow highlight indicates the correct response 

A 

0 

3 

8 

14 

21 

15 

2 

6 

3 

21 

7 

2 

4 

0 

22 

20 

5 

7 

B 

20 

10 

1 

4 

2 

3 

4 

1 

10 

0 

15 

5 

8 

1 

1 

1 

9 

4 

c 

0 

3 

4 

3 

1 

0 

1 

2 

5 

0 

1 

13 

5 

18 

0 

1 

5 

0 

D 

4 

7 

10 

2 

0 

6 

17 

15 

6 

2 

0 

2 

5 

4 

0 

1 

4 

11 

other 

0 

1 

1 

1 

0 

0 

0 

0 

0 

1 

1 

2 

2 

1 

1 

1 

1 

2 


Percentage for each question 


Yellow: no concern, red: concern (criteria 60%) 


0.21 

0.25 


A 

0.00 

0.13 I 0.35 

0.61 

0.88 

0.63 

0.08 

0.25 

B 

0.83 

0.04 

0.17 

0.08 

0.13 

0.17 

0.04 

c 

0.00 

0.13 I 0.17 

■ 

0.04 

0.00 

0.04 

0.08 

D 

0.17 

0.30 

0.09 

0.00 

0.25 

0.71 

0.63 


0.91 

0.00 

0.00 

0.09 


0.09 

0.23 


0.30 
0.65 
0.04 

0.00 I 


0.18 

0.36 



0.00 

0.04 

0.78 

0.17 


0.96 

0.04 

0.00 

0.00 


0.87 

0.04 

0.04 

0.04 


0.22 

0.39 


0.32 

0.18 



0.17 


Table 3. Analysis of respondent answers to Academic Knowledge Test 1 
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Question 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 


Number respondents for each answer Yellow highlight indicates the correct response 

A 

5 

6 

15 

10 

4 

0 

1 

1 

18 

18 

2 

4 

14 

4 

5 

22 

13 

0 

B 

3 

16 

3 

6 

12 

21 

3 

19 

2 

0 

1 

6 

1 

5 

1 

1 

3 

9 

c 

14 

1 

2 

6 

3 

0 

2 

0 

2 

2 

16 

12 

1 

13 

2 

0 

2 

2 

D 

1 

0 

3 

1 

4 

1 

17 

3 

1 

2 

4 

0 

5 

1 

15 

0 

5 

12 

other 

1 

1 

1 

1 

1 

2 

1 

1 

1 

2 

1 

2 

3 

1 

1 

1 

1 

1 


Percentage for each question 


Yellow: no concern, red: concern (criteria 60%) 


A 

0.22 

B 

0.13 

c 

0.61 

D 

0.04 



Table 4. Analysis of respondent answers to Academic Knowledge Test 2 


c. Analysis 

The design of the paired before and after surveys dictated an 
analysis of paired Likert scale responses. That is, we looked at the differences 
between numerical responses to the attitude questions and skill sets for changes 
in attitude or proficiency. We first analyzed the survey data to find any linear 
dependence (multicolinearity) between the survey questions that would indicate 
that the answer to a specific question would influence the answer to one or more 
others. To evaluate this we used a pair plot of the responses to all questions on 
the “after” survey, as shown in Figure 9. Not surprisingly, this pair plot showed a 
strong linear relationship between all four questions which indicated that they 
together shared some attribute(s) of the serious gaming that the participants felt 
would be valuable in unit training. We found similar linear relationships to be 
strong in other questions pertaining to self-assessment. 
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Figure 9. Pair plot of survey questions related to user appreciation of 

simulation training 

Next, we used a Wilcoxon Rank test to determine which responses 
had changed with statistical significance. For the MOVES students who trained 
in the first part of the study, we found three tactical skills to have significance as 
shown in Table 5. The MOVES students, representing untrained personnel in a 
non-Marine setting with respect to convoy experience, responded that VBS 2™ 
improved knowledge and ability of immediate action to an unblocked ambush, 
cordon and 360°security, and casualty evacuation. 
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4 

Unblocked 

Ambush lA 

-2.9266 

0.0017 

5 

Cordon and 360 
degree security 

-2.0991 

0.0179 

10 

Casualty 

evacuation 

-2.1297 

0.0166 


Table 5. Self-assessment questions that showed a significant difference in 

response for MOVES students 

When we conducted the similar exercise at WFTBN and a group of 
Marines with mixed convoy experience was tested, we found three tactical skills 
to have significant change, as shown in Table 6. None of the skills was common 
to those that showed significant change in the MOVES group. It is important to 
note that these results were only significant for Marines who had little or no 
convoy experience. The Marines who had more than twenty-five convoys found 
that the training provided little or no value and actually degraded their self- 
assessed performance in some cases. 


# Question z p-value 


1 

React to 

unexploded lED 

2.1503 

0.0158 

2 

React to 

detonated lED 

2.0827 

0.0186 

6 

Employ weapons 

2.1333 

0.0164 


Table 6. Self-assessment questions that showed a significant difference in 

response for WFTBN Marines 
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4. 


Discussion 


a. Initial Findings 

The series of initial pair plots revealed strong linear relationships 
among aspects of the game experience, as well as the participants’ assessment 
of their level of skill and familiarity with convoy tactics. These strong linear 
relationships led us to investigate what aspects of the game, or tactical skills, 
affected the Likert scaled responses most. Having had each participant answer 
detailed pre- and post-training surveys, we then made a data set that enabled us 
to review how the delta of each question was affected by the training. 

The Wilcoxon Rank tests showed that VBS 2™ did not significantly 
change participants’ attitudes or appreciation of serious gaming. This is an 
interesting, if not surprising, result, given that the relative youth of our participants 
did not translate into excitement or increased appreciation for serious gaming. 
The resulting p-value of this assessment meant that there was no basis to reject 
our first null hypothesis: The use of VBS 2™ does not change the appreciation 
of serious games as a training tool as measured by the Likert scaled responses 
of participants. 

b. Digging Deeper 

The failure to reject our first null hypothesis did not necessarily 
mean that VBS 2™ was devoid of any benefits toward training. We examined 
self-assessed proficiency to determine if participants felt that their tactical abilities 
had changed even if their opinion of simulation as a training device had not. 
Knowing that our participants would have various military skills, experiences and 
time in the military, we structured the pre- and post-training surveys to 
incorporate these factors. By cross-examining military experience with questions 
that elicited our participants to assess their personal proficiency in immediate 
action drills (such as their confidence/ability to take action against a blocked 
ambush), we obtained two revealing solution sets to Wilcoxon Rank tests: 
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• MOVES students who reported themselves as having little 
convoy knowledge and experience reported that VBS 2™ 
improved their knowledge, ability and confidence to take 
immediate action in the cases of an unblocked ambush, 
cordon, and establish 360° security and conduct casualty 
evacuation. 

• Among the operational Marines who participated in the 
testing at WFTBN, those who reported having no practical 
convoy experience felt that VBS 2™ improved their 
knowledge, ability and confidence to take immediate action 
in the cases of an unexploded lED, detonated lED, and in 
the tactical employment of weapons. 

These tests suggest that VBS 2™ might be an effective introductory 
training tool for those who have little or no convoy experience. Where actual 
convoy rehearsals and training may be difficult or burdensome, VBS 2™ seems 
to be a clear alternative to expose junior Marines to training at the schoolhouse 
or first duty station. 

We examined the six skills for which participants identified an 
increase in self-assessed proficiency. We could find no reason that would 
explain why the MOVES group showed a change in three skills and the WFTBN 
novice users showed a change in three separate skills. The groups certainly 
differed, but the differences did not relate to the skills in any way. From 
observation of the training in progress, we offer that participants might tend to 
indicate a change in self-assessed proficiency for the exercises that go well. 
Some of the exercises experienced difficulties, such as communications 
problems, computer glitches, and internal command and control problems. On 
the other hand, other parts of the exercise went smoothly so that standard 
procedures were executed as intended. We suggest that the particular skill 
assessed as improved might relate more to what goes well than to any other 
factor. If novice trainees see a smooth execution, then they may feel more 
proficient in that particular skill. 
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c. 


Applicable for Even the Veteran Marine 


For the veteran Marines who reported that VBS 2™ may have 
actually degraded their skill or confidence in their convoy ability, we recommend 
further study and/or examination of the following factors: 

• VBS 2™ was not employed in the context that a veteran 
Marine might see with his actual unit. Such a future 
occasion might entail them virtually driving the same convoy 
route prior to the actual convoy, where speeds, terrain 
features and distances would all serve as useful, albeit 
virtual, exposure to the actual route. This, in essence, would 
afford them to “conduct the convoy prior to conducting the 
convoy.” 

• Those who reported a degradation in skill may have actually 
only been made aware of their “rustiness” or inflated sense 
of personal mastery of what it is to conduct a convoy. In this 
sense, perhaps VBS 2™ serves as a very useful reality 
check. 

To this end, the results of the extended study of the responses to the 
pre- and post-training surveys led to our conclusion that, in certain cases, we could 
reject our second null hypothesis. Participants feel that the use of VBS 2™ does 
not change their small-unit convoy knowledge and skills as measured by Likert 
scaled responses. The instances where the Wilcoxon Rank tests yielded p-values 
supporting such rejection of the null hypothesis can be summarized as those 
involving either novice convoy trainees or exposure to immediate action drills. 

d. Practical Tips for Future Studies 

VBS 2™ is one of many in a wide spectrum of serious games that 
have either already been fielded or are currently in development. The use of 
VBS 2™ and other similar serious games as a device to expose junior 
servicemen to tactical scenarios is clearly fertile ground for further study. 

In addition to determining how VBS 2™ affected participant 
knowledge and confidence, these two studies also provided an opportunity to 
investigate the practical aspects of conducting training using the DVTE suite. 
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The Marine Corps intends DVTE, including VBS 2™, to be a portable resource 
for every infantry battalion. Because VBS 2™ is intended to be a portable 
resource, we found that it will be imperative to every unit to appreciate the 
following: 

• Participants must take the training seriously. Negligent 
discharges and fratricide severely hinder the effectiveness 
and spirit of the training. 

• The length of training and the days on which it is scheduled 
(WFTBN training occurred on Friday) ought to be carefully 
considered to enable participants to more fully focus on the 
training at hand. 

• The current communications package (JVTR) is a burden. 
Possible replacement, enhancements, or unit workarounds 
need to be considered. The optimal solution is an internal 
communication package within VBS 2™ itself. 

• A larger library of terrain needs to be developed quickly. 
Areas such as Twentynine Palms need improved fidelity. 
The size of the Twentynine Palms database makes it 
unwieldy. For small unit tactical exercises, fidelity is more 
important than terrain size. 

• The computers’ operating systems have recurring glitches 
that will need to be recorded in order to develop patches or 
user-level workarounds that will support continued training. 

e. Conclusion 

The objective of this research was to evaluate user perceptions of 
the serious game VBS 2™. As a pilot study, it was also the first effort in having 
multiple people work together in a common virtual environment. We met both 
objectives and, moreover, conclusive data led us to the fact that in this case, a 
broad hypothesis could not be applied to our diverse sample population. 
Through data analysis and cross-examination of several key factors (i.e., age, 
rank, convoy experience), we were able to determine that the perceived 
effectiveness of VBS 2™ as a training tool was much larger than a single Likert 
scaled response. In fact, as a training tool, it holds the potential to be applicable 
to junior and veteran servicemen alike, albeit in different ways. 
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IV. EXPERIMENT IN SUPPORT OF HYPOTHESIS 1 


A. INTRODUCTION 

Experiments 1, 2, and 3 prepared us to use VBS 2™ to train Marines. 
Experiment 1 demonstrated sound practices to familiarize users with the VBS 2™ 
interface so that they could use the simulation as an extension of their 
warfighting skills. Experiments 2 and 3 provided a forum to test scenarios to 
determine what worked and what did not. Armed with this information, we could 
address the questions of this thesis project. With this in mind, we designed 
Experiment 4 to address the first thesis research question: 

• Does preliminary training with VBS 2™ positively impact performance 
in a platoon-level tactical scenario? 

In our consideration of the task of training convoy operations with virtual 
environments, we noted that different approaches could influence the 
effectiveness of the training. A study at the Israeli Air Force flight school in 1993 
inspired our choice of approach. In that study, cadets used a computer game 
called Space Fortress II in the early stages of flight training. One group played 
seven part-task games, one at a time, receiving feedback after each. The other 
group used a full mission game with all tasks covered in the same game. A 
control group trained with traditional techniques. The study found that both 
simulation groups performed better in subsequent real-world flights than the 
control group, and the part-task group outperformed the full mission group 
(Gopher et al., 1994). For our work, we considered two different arrangements of 
the skill sets to be trained. In one application, a Marine trained each skill 
individually in depth, presenting progressively more difficult training for each skill 
individually before moving to the next skill. The second approach combined all 
the skills into a single training evolution and developed progressively more 
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difficult training for all the skills as a group. We were interested in whether one of 
these approaches would produce better results, so we developed the second 
research question for Experiment 4: 

• Is there a difference between using a full mission approach versus 
a part-task approach to training when using VBS 2™ for platoon- 
level tactical training? 

In considering the design of this experiment, we relied heavily on previous 
student work at the Naval Postgraduate School. The previously mentioned work 
of U.S. Army Majors J. Nolan and J. Jones with Delta Force: Black Hawk Down- 
Team Sabre™ served as a basis for studying how gaming technology could 
support training goals (Nolan & Jones, 2005). Likewise, Marine Corps Major Neil 
Fitzpatrick and Turkish Army Captain Limit Ayvaz worked with a decision-making 
trainer called Close Combat Marine, and their work provided insight into Best 
Practices for leveraging surveys and questionnaires to gain insight into user 
performance (Fitzpatrick III & Ayvaz, 2007). We summarized the body of work 
done by the Aussie Virtual Environments & Simulation Laboratory (VESL) with 
VBS 2™ in Chapter II of this thesis. All of these research efforts directly 
contributed to the design of Experiment 4. 

In fact. Experiment 4 is similar enough to some of these previous efforts 

that it may seem like just another repetition of the same work. For Experiment 4, 

we felt it was crucial to conduct the study with an operational unit preparing for 

real-world contingencies. We sought to take the personal computer-based game 

beyond the walls of the schoolhouse and the clear-cut environs of the university 

laboratory. Experiment 3, with its work with infantry Marines at Weapons and 

Field Training Battalion, demonstrated that enlisted Marines with a “day job” in 

the operational forces might perceive personal computer-based game training 

differently. For this reason, we expended much effort to establish an experiment 

in the operational forces. With this in mind, two competing factors drove 

Experiment 4: the scientific rigor associated with any hypothesis test and the 

training needs of an operational unit very much involved in preparing to fight a 

real-world war. Throughout the research project, we placed equal emphasis on 
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both concerns. That is, we sought soundly and scientifically to gain insight into 
the thesis question at hand, but we also sought to train Marines as best we could 
with the tool used for the study. These Marines needed the training we offered, 
and failure to provide it was unacceptable. 

We chose a reserve Marine infantry battalion preparing for deployment to 
combat theater for Experiment 4. In part, timing drove this decision, because this 
particular battalion happened to be in the preparation phase at the right time for 
this research project. However, other factors drove this decision as well. 
Reserve Marines typically do not enjoy the benefits of simulation available to 
active duty Marines. Reserve units dot the U.S. map with no regular proximity to 
active duty training installations. While reserve units are eligible for the 
Deployable Virtual Training Environment (DVTE) described earlier, they have a 
lower priority for the equipment than their active duty counterparts. In fact, this 
particular unit had been working to get the gear prior to their deployment, but at 
the time they left home site, no DVTE equipment had been used. Because 
reserve units come from a single geographic location, they typically enjoy better 
personnel stability than active duty units. Additionally, reserve Marines often stay 
in the same platoons and companies for their full enlistment, and some may 
spend entire careers in the same unit. We viewed this unit cohesion as a 
positive contributor to our research plans. This symbiotic relationship of a unit 
with a training need and a research project with a need for a unit promised 
mutual benefit. 

In the end, we developed an experiment design that was a compromise 
result of the competing needs of science and operational training. We conducted 
our research with one of the battalion’s infantry companies. We conducted the 
simulation training at a phase in the battalion’s pre-deployment preparations in 
which the unit had deployed from home site to Camp Pendleton, California. We 
had a week of training “white space,” or otherwise unobligated training time, to 
conduct training in support of our project. During this week, platoons conducted 
training in support of the study for four hours per day for a four-day training week. 
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The particular company selected for the research had a live exercise aboard 
Camp Pendleton scheduled that would serve as an evaluation metric for the 
study. Within these parameters, we developed Experiment 4 using lessons 
learned from the work mentioned above. 

B. METHOD 

1. Participants 

The Weapons Company of the reserve infantry battalion participated in 
Experiment 4. For convoy operations, the Weapons Company was already 
divided into three Motorized Assault Platoons (MAPs), and these MAPs served 
as the three treatment groups for the experiment. A Marine Corps Weapons 
Company includes infantry with a variety of specialized skills, including heavy 
machine guns, anti-tank missiles, medium mortars, and other heavy weapons 
organic to an infantry battalion. Many of these weapons, such as the 0.50 caliber 
machine gun, the MK 19 automatic grenade launcher, and the Tube-launched 
Optically-tracked Wire-guided (TOW) missile are often vehicle-mounted for 
infantry battalion operations. Therefore, mounted operations, including the 
convoy mission of moving men and materiel from one place to another, were 
standard routine training objectives for the Weapons Company. Moreover, 
proficiency in this domain would certainly be a demand upon deployment to 
combat theater. 

Most of the Marines in each MAP were infantry, including machine 
gunners (MOS 0331), mortarmen (MOS 0341), and anti-tank missilemen (MOS 
0352). However, at the time of the experiment, the ranks of the infantry battalion 
were being fleshed out with a detachment from a communications battalion, so 
each MAP had a few communicators, including field radio operators (MOS 0621) 
and ground communications repairmen (MOS 2844) sprinkled into the platoon 
organization. Each MAP varied in size, but twenty-four Marines from each MAP 
participated in the study. Unit leadership randomly chose these Marines with no 
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input or cognizance on the part of the research team. The unit emphasized intact 
vehicle crews participating in the study so, for the most part, the participants 
were vehicle crew members identified at home site by the Company Commander 
well prior to the execution of the experiment. With this in mind, we assumed that 
there were no biases in the selection of the training audience; rather, the 
Company Commander crafted the training audience from the stable crews early 
in the planning process. 

Each MAP had a Lieutenant serving as the Platoon Commander, and for 
Experiment 4, these Marines served as the trainers. In this capacity, they 
organized the participants, supervised exercise briefing, participated in using the 
simulation exercise to meet specific platoon and company training goals, and 
conducted the debriefing after each exercise. The Platoon Sergeant for each 
MAP, the senior enlisted Marine in the unit, did not participate in the training. All 
three MAPs were larger than the twenty-four man training audience 
accommodated in the virtual training, so the Platoon Sergeant handled the 
Marines who were not being trained in the experiment. Each MAP had an 
enlisted Marine who served as the Convoy Commander for the exercise. For 
MAPs 1 and 2, the Convoy Commanders were senior Sergeants (E-5), and for 
MAP 3, the Convoy Commander was a junior Staff Sergeant (E-6). 

Most of the Marines who participated in the study were young Lance 
Corporals (E-3). Excluding the aforementioned Convoy Commanders, MAP 1 
had three noncommissioned officers (NCOs); MAP 2 had six NCOs; and MAP 3 
had six NCOs. The average age of the participants was 23.8 years. Using 
Analsyis of Variance (ANOVA) and paired t-tests, we noted that there was a 
difference in age between MAPs 2 and 3 (p-value = 0.0164) with MAP 2 having a 
mean age of 22.75 years and MAP 3 having a mean age of 25.3 years. This 
difference arose from a few individuals and was not a platoon-wide trend. While 
no one in MAP 2 had reached their 30s, MAP 3 had a 31-year-old Staff Sergeant 
and a 35-year-old Sergeant who swayed the average. The mean time in service 
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for all participants was approximately three-and-a-half years and the mean time 
in unit was approximately two years. There was no significant difference in 
platoons for these attributes. 

2. Apparatus 

The project used twenty-six Dell Precision M6300 laptops from a suite of 
the Marine Corps Deployable Virtual Training Environment (DVTE) package. As 
in previous experiments, computers were networked using D-LINK DGS-2208 8 
port switches. Peripheral equipment, including mice, cables, and switches, came 
from the standard DVTE package. Unlike Experiments 2 and 3, the network 
configuration for the familiarization training was the same as for the training 
exercises. We networked all twenty-six computers together using seven 
switches, allowing all machines to participate together in the same environment. 

The Weapons Company occupied barracks in Camp Pendleton so that 
they had access to a parade deck and large classroom. We set up the twenty-six 
computers in the classroom, networked them, and left them configured for the 
week. The classroom was a large facility configured for press briefings so that 
we had easy access to overhead projection and power at individual seats. We 
used the back six rows of tables in the room. Each table row had eight seats, 
and we used the seats to maximize the space between participants as much as 
possible. The three participants in a vehicle crew sat in a single row, with two 
rows having two vehicle crews (six computers) and the remaining four rows 
having a single vehicle crew each. For the majority of the training, the trainer 
and exercise controller each had a computer and sat facing the Marines 
conducting the exercise. In this way, both the trainer and the controller could 
monitor the virtual action on the screen while observing the action of the 
participants in the room. Figure 10 shows the room configuration. 
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Figure 10. Simulation training site for operational unit at Camp Pendleton, CA 


As in Experiments 2 and 3, participants communicated using the Joint 
Virtual Tactical Radio (JVTR), a voiceover IP system. From the lessons learned 
in Experiments 2 and 3, we attempted to reduce the complexity of the 
communications problem by using only a single radio net. Only the vehicle 
commanders used the radio to talk on a single net. The trainer monitored the 
platoon traffic using his computer and participated as higher headquarters on the 
platoon’s single tactical net. Participating Marines were taught nothing about the 
radio’s operation except how to push a key to talk. 

The control group conducted training consisting of sand table exercises 
and garrison rehearsals. This platoon used the ample sandy areas outside the 
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barracks for sand table exercises. They conducted garrison rehearsals using the 
unit’s vehicle on the parade deck outside the barracks. The platoon commander 
controlled the exercises. 

As mentioned for Experiments 2 and 3, Marines typically operate vehicles 
in teams of three: a vehicle commander riding in the passenger seat, a driver, 
and a gunner for the vehicle’s weapon system. For all tactical scenarios, we 
grouped participants in this way. Eight vehicle crews participated in the exercise. 
The vehicles were all HMMWVs with a mix of 0.50 caliber machine guns, MK 19 
automatic grenade launchers, and TOW missiles that reflected how the MAPs 
trained in the real-world according to unit rosters. 

As in Experiments 2 and 3, we used statistical group comparison 
techniques including the paired t-test for analysis. For all statistical tests, we 
used an alpha level of 0.05. 

3. Procedure 

As in Experiment 3, the experiment focused on training four tactical 
convoy tasks: reaction to an lED, reaction to ambush, vehicle recovery, and 
casualty evacuation. We chose to evaluate the effectiveness of the training in 
three separate domains. First, a set of knowledge tests was used to determine if 
the training affected participants’ academic knowledge of the skill sets. Second, 
surveys were used to determine how participants assessed their own proficiency 
and their unit’s proficiency in each skill set. Finally, the live exercise served as 
an indicator of the participants’ practical application of the skills in the real world. 

We used the same knowledge tests that had been used in Experiment 3 
with Weapons and Field Training Battalion. These tests consisted of eighteen 
multiple choice questions each. The questions on the two tests were paired. In 
this way, the tests were different, but each test had the same number of 
questions in each domain. For example, if Test 1 had a question about 
unblocked ambush. Test 2 had a similar question. Data from Weapons and Field 
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Training Battalion was used to ensure the tests were equally difficult. 
Participants took one of the tests before training and the other test after training. 
To further avoid testing bias, half the participants in each group took Test 1 first 
and Test 2 second and vice versa for the other half. The tests are included in 
Appendices P and Q 

The survey portion of the experiment also used the same paperwork that 
was devised for Experiments 2 and 3. The three-page initial demographic 
survey, included as Appendix M, featured several questions about the individual, 
such as age, rank, time in service, computer usage, and convoy experience. 
Another set of questions intended to determine the participants’ attitudes toward 
personal computer-based simulation training. The next set of questions 
determined the participants’ perception of tactical skill in the trained skill sets. As 
an addition to the Experiment 3 survey, a follow-on set of questions determined 
the participants’ perception of their unit’s tactical skill (that is, a person may 
believe he does not understand how to react to an lED because he is novice, but 
that the platoon as a whole is trained in that skill). Responses to all attitude and 
training proficiency questions were recorded on a five-point Likert scale. After 
the training, participants completed a survey with the same simulation attitude, 
individual training proficiency, and unit training proficiency questions that were on 
the demographic questionnaire. They also had a page to record subjective 
comments as answers to several open-ended questions about what they liked 
and did not like about the training. The questionnaires were the same for 
simulation groups and for the control group, with the following exceptions. On 
the post-training questionnaire, the control group did not have computer attitude 
questions. Also, on the post-training questionnaire, the simulation groups 
answered a set of Likert scale response questions addressing their simulation 
experience. The post-training survey for the simulation groups is included as 
Appendix N, and the post-training survey for the control group is included as 
Appendix O. 
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The live exercise was a portion of a regularly conducted training package 
designed for reserve infantry battalions training at Camp Pendleton. This 
battalion was the sixth to complete such training. The training package consisted 
of an urban patrolling component and a vehicle convoy component. The vehicle 
convoy portion consisted of two days of repetitively conducting a short three-mile 
course and three days of operating on a longer twenty-mile course. The short 
course included the following evaluated skill sets: establishment of a snap 
vehicle checkpoint (VCP), reaction to an lED, and reaction to lED-triggered 
ambush. Because any of the events could trigger a vehicle or personnel 
casualty, the exercise adequately served as a mechanism for observing the 
trained skills in practice. A cadre of three Marines graded the short convoy 
course. All of the training cadre had graded the same course for other 
companies in the battalion. Evaluators graded each vehicle on the course using 
a three-point rating of untrained (U), trained (T), and mastery (M). The evaluator 
grading system was developed by adding the three-point grading scale to the 
debrief form already in use for the exercise, and the resulting evaluation form is 
included as Appendix S. All evaluated portions of the course were positioned in 
terrain so that evaluators could observe the unit’s activity from high overwatch 
positions or from within the convoy itself. 

The simulation training encompassed the two separate treatments of full 
mission and part-task training. VBS 2™ offers several terrain databases, and we 
used two separate databases to give the Marines variety in their training 
experience. We used the Twentynine Palms database for initial training including 
the interface familiarization exercise. The Twentynine Palms database is a 
virtual replication of the Marine Corps’ training areas at the Marine Corps Air 
Ground Combat Center in Twentynine Palms, California. This terrain features 
open desert with mountains dividing the area into distinct corridors. The terrain 
covers a large space more than 60 kilometers east to west and more than 40 
kilometers north to south. For the more difficult training exercises, we used the 
Sahrani database. Sahrani is an imaginary island developed for gamers that 
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provides a Mediterranean feel. The terrain is hilly and wooded in many places. 
Rural settlements dot the landscape. The terrain area is much smaller, featuring 
only about 15 kilometers at its widest point from east to west; the whole database 
covers only 400 square kilometers. 

Both simulation groups started training by completing the informed 
consent paperwork, demographic survey, and initial knowledge test. They then 
completed an hour of interface familiarity training based on the work from 
Experiment 1. Afterward, they completed a short exercise called “Training 
Wheels” simply to familiarize them with the idea of working together in a vehicle 
convoy, using platoon communications, and coordinating movement and fires. 

After the initial paperwork and training, MAP 1 participated in the full 
mission training. Throughout the four days, MAP 1 completed five convoy 
scenarios in the Twentynine Palms database and two scenarios in the Sahrani 
database. Each of the scenarios involved all four skill sets in question. In 
general, MAP 2 concentrated on casualty evacuation the first day, vehicle 
evacuation the second day, reaction to ambush the third day, and reaction to lED 
the final day. The final exercises did not involve true isolation of the skill sets; for 
example, participants in a reaction to ambush scenario might have to evacuate a 
casualty at some point because of the course of events. Both platoons executed 
the same final full mission exercise in Sahrani terrain at the end of the training 
evolution. Table 7 shows the details of the training plan for both platoons. 
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Preparatory training for aii 

Familiarity _ Interface training package with practice exercise _ 

Training Wheels | Convoy movement practice 

Fuii mission training scenarios 

Level 1 Noble Inert lED, Ambush 2x fire team (FT), Squad with inert lED 

_ Pass _ and single vehicle and personnel casualty _ 

Level 2 Gays 1x FT, 1x Squad, 1x FT with daisy chain I ED and vehicle 

_ Pass _ and personnel casualty, civilian farm _ 

Level 3 Rainbow 2x trucks, 1x Squad with lED, civilian farm, 1x FT with daisy 

Canyon chained lED and vehicle and personnel casualty, 1x FT, 

_ civilians _ 

Level 4 Sahrani 1x RPG dismount with live lED, 2x RPG dismounts, 1x FT, 

Iguana 1x Squad with daisy chained lED, 1x RPG dismount, 
_ intermittent traffic and civilian activity _ 

Part-task training scenarios 


Task 

Level 1 

Level 2 

Level 3 

Level 4 

Level 5 

Level 6 

Casualty 

Trng 

Wheel 

Casualty 

Noble Pass 
Casualty 

Gays 

Pass 

Casualty 

2x 

casualty 

Cayo 

Casualty 

2 sep cas 
FT / RPG 

Flunapu 

Cas Pers 

3x 

casualty 

FT + Truck 

Cresta 

Free 

Play 

1 Truck 
Place 

cas 

Vehicle 

Trng 

Wheel 

Casualty 

Noble Pass 
Casualty 

Rainbow 

Casualty 

2x 

casualty 

Flunapu 
Cas Veh 

2 sep cas 

RPG/Tru 

ck 

Cresta 

Casualty 

2x 

casualty 

FT 

Flunapu 

Free 

Play 

1 Truck 
Place 

cas 

Ambush 

Trng 

Wheel 

Ambush 

Noble Pass 
Ambush 
lx FT targ’t 

Gays 

Pass 
Ambush 
lx FT 
inert 

Flunapu 

Ambush 

2x FT 

Live 

Cayo 

Ambush 

Squad+FT 

Cresta 

Free 

Play 

2x Truck 

lED 

Trng 

Wheel 

Casualty 

Noble Pass 
lED 

Undefended 

Rainbow 
lED 
lx lED 
RPG 
Team 

Cayo 

lED 

2 sep lED 

RPG 

each 

Cresta 

lED 

lED+Chain 

RPG/2X 

FT 

Flunapu 

Free 

Play 

2x Truck 

Place 

lED 


Evaiuation exercise for aii 

Sahrani Ixel 1x FT, 1x FT with daisy chained lED and personnel casualty, 

2x trucks, 1x Squad with daisy chained lED and personnel 
_ casualty, 1x sniper, intermittent traffic and civilian activity _ 

Table 7. Convoy training matrix for simulation groups 
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For both simulation training groups, the exercises increased in complexity. 
We varied this complexity in different dimensions. First, the exercises had 
varying degrees of assistance. All exercises included a map, and that map may 
have provided a location and description of the problem the unit would react 
against, just a location of the problem, or no information about the problem at all. 
For early exercises, an overhead view of the exercise area was projected for the 
participants so they could see a macro-level view of their actions and the actions 
of others. Second, the exercises had varying levels of collateral activity. Initial 
exercises featured nothing other than the exercise problem itself. More difficult 
levels included civilians, cars and traffic, animals, and buildings and built up 
areas. Third, the problem difficulty increased within each skill set. More difficult 
scenarios involved reacting to multiple casualties instead of just one; recovering 
multiple vehicles instead of just one; reacting to chained lEDs, fake lEDs, and 
remotely detonated lEDs; and complex ambushes with integrated Rocket 
Propelled Grenade (RPG) gunners and potential crossfire situations. Enemy 
activity was more coordinated in advanced exercises, requiring convoys to deal 
with more than one problem at once (e.g., reacting to ambush while evacuating a 
casualty). Advanced exercises also incorporated some enemy indirect fire. 

The control group participated in exercises that mirrored a combination of 
the part task and full mission groups’ activities. To mirror the part-task activities, 
the platoon conducted immediate action drills with actual weapons and vehicles on 
the local parade deck. Like the part-task group, the control group concentrated on 
a single skill set per day. To mirror the full mission activities, the control group 
conducted sand table exercises that were similar to the full mission scenarios 
executed in simulation. The platoon commander used the full mission simulation 
materials to develop his sand table training. 

C. RESULTS 

The study involved three different measures of effectiveness: knowledge 
tests administered before and after training; surveys administered before and 


111 



after training; and a live exercise conducted after the training. Only the surveys 
produced significant results. We will discuss the results from each of these 
evaluation mechanisms in turn. 

1. Knowledge Tests 

We graded the multiple choice tests that were taken before and after the 
training and recorded the answers in a spreadsheet to analyze differences in the 
two tests. We summarized each participant’s performance with the number of 
correct answers produced on each test. We then looked at the simple difference 
between correct answers on the before test versus the after test. We used 
ANOVA to test whether there was a difference in the mean performance of each 
platoon; no significant difference resulted. We also conducted t-tests on the 
performance of each platoon separately to test the hypothesis that participants’ 
academic knowledge of convoy operations improved after training. No platoon 
performed better on the knowledge test with significance. 

We tested to see if the tests were truly equal in difficulty. First, we looked 
at participant performance on each 18-question test regardless of whether the 
participant took the test before or after training. On average, participants 
answered 9.87 questions correctly on Test 1 and 9.09 questions correctly on 
Test 2. Next we looked at participant performance when taking each test before 
training and then after training. On average, participants taking Test 1 before 
training answered 9.33 questions correctly, while participants taking Test 1 after 
training answered 9.86 questions correctly. Participants taking Test 2 before 
training answered an average of 9.18 questions correctly, while participants 
taking Test 2 after training answered an average of 9.10 questions correctly. For 
both ways of looking at the issue, there was no significant difference in the 
number of questions answered correctly, so no evidence suggested a significant 
difference between the two tests. 

Finally, we investigated whether participants responded to the simulation 
attitude questions based on any specific biases from the demographic 
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information. We used correlation and regression analysis to check each of the 
demographic categories and found no relation to the simulation attitude results. 
In summary, the practical application training, regardless of whether it was 
simulation, traditional, full mission, or part task, resulted in no change in 
academic performance on the convoy tests. 

2. Surveys 

a. Individual Attitude toward Simulation 

Participants who completed the simulation training took surveys 
before and after training in which they assessed their attitude toward simulation 
for training. They answered four questions using a five-point Likert scale grading 
system. We used this construct to analyze whether their attitude toward 
simulation changed as a result of the training by subtracting the difference 
between the before and after ratings for each skill. We analyzed these 
differences to determine whether the different types of simulation training 
produced different changes between the two platoons. One participant in the 
part-task platoon and one participant in the full mission platoon failed to complete 
the full week of training, so their initial surveys were not considered in this paired 
comparison. In total, responses were analyzed from forty-six participants. 

First, we combined the four questions into a single composite score 
by averaging the answers to each question. We used a t-test to test the null 
hypothesis that the platoons did not differ in their change in attitude toward 
simulation. With an F-ratio of 7.2142 and a p-value of 0.0102, we found that the 
two platoons did differ with the full mission simulation trained platoon having a 
higher assessed attitude than the part-task simulation trained platoon. With this 
in mind, we examined more closely by performing the same analysis on each 
question individually. We found that the platoons only differed for two of the four 
questions. The full mission simulation trained platoon was higher for “Computer- 
based simulation is an effective training tool” and “I think a unit should use 
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computer-based simulation in its tactical training.” The platoons did not differ for 
“Today’s planned training will improve my ability to conduct convoy operations” 
and the mirror question for the unit’s ability. Table 8 lists the data for this analysis. 


# 

QUESTION 

F-RATIO 

P(>F) 


COMPOSITE - Attitude toward simulation 

7.2142 

0.0102 

1 

Computer-based simulation is an effective training 
tool. 

6.7368 

0.0128 

2 

Today's planned training will improve my ability to 
conduct convoy operations. 

2.0873 

0.1556 

3 

Today's planned training will improve my unit's ability 
to conduct convoy operations. 

1.2564 

0.2684 

4 

1 think a unit should use computer-based simulation in 
its tactical training. 

8.6139 

0.0053 


Table 8. Analysis of before and after responses to statements related to attitude 
toward personal computer-based training; yellow highlight indicates 
significant difference in attitude between MAP 1 and MAP 2 


Next, we analyzed the questions by platoon, as shown in Table 9, 
to see how each platoon’s attitude toward simulation changed as a result of the 
exercise. We calculated a difference between Likert scale marks for each 
question before and after training. We then used a t-test to investigate the null 
hypothesis that the platoon’s attitude toward simulation did not change. For the 
full mission simulation trained platoon, we found no evidence to reject the null 
hypothesis. However, for the part-task simulation-trained platoon, attitudes 
toward simulation went down for all but “Today’s planned training will improve my 
ability to conduct convoy operations.” 


Question 

MAP 1 

tTest Prob>|t| 

MAP 2 

tTest Prob>|t| 

Composite 

1.0675 

0.2973 



1 

1.4316 

0.1663 

Kfl 


2 

1.0000 

0.3282 

-1.0446 

0.3075 

3 

-0.4393 

0.6647 



4 

1.2981 

0.2077 

r-..- 



Table 9. Analysis of before and after responses to statements related to attitude 
toward personal computer-based training by platoon; red highlight 
indicates significant decreased attitude 
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In summary, the two simulation-trained platoons did differ in how 
their attitudes toward simulation changed as a result of the training. Specifically, 
they differed in their attitude toward simulation in general, while there was no 
difference in their attitude toward the training they received in this particular set of 
training sessions. There was no evidence that the simulation training changed 
the attitude of the full mission simulation trained platoon, but the part-task 
simulation trained platoon’s attitude toward simulation got worse. 

b. Individual Tactical Proficiency 

Participants took surveys before and after training. On these 
surveys, they assessed their proficiency and the proficiency of their unit as a 
whole in a series of skills using a five-point Likert scale grading system. With this 
construct, we analyzed self-assessed proficiency by subtracting the difference 
between the before and after ratings for each skill. We then analyzed these 
differences to determine differences between platoons and perceived 
improvements in performance. One participant in the part-task platoon and one 
participant in the full mission platoon failed to complete the full week of training, 
so their initial surveys were not considered in this paired comparison. In total, 
responses were analyzed from seventy participants. 

We combined some of the questions to create composite scores by 
skill group. We created one composite score encompassing all thirteen tactical 
skills as a group. Additionally, we created skill group composite scores including 
the combination of two questions addressing reaction to lED; two questions 
addressing reaction to ambush; three questions addressing weapons 
employment; and three questions addressing communications. Vehicle recovery 
and casualty evacuation were evaluated as stand-alone questions. We 
developed the composite scores for the skill groups by averaging the responses 
from the questions within the group. 

We used ANOVA to determine whether the platoons differed in their 
perceived tactical proficiency before and after training, as shown in Table 10. We 

115 



found that the platoons did differ based on the overall composite tactical score with 
an F-ratio of 7.95 and a p-value of 0.0008. Using pairwise t-test comparisons, we 
found that both MAP 1 and MAP 2 differed from MAP 3, but that they did not differ 
from each other. That is, both simulation-trained groups felt that their proficiency 
increased after training, but the traditionally trained group did not. 


# 

QUESTION 

F RATIO 

P(>F) 

All 

COMPOSITE INDIV - Tactical 

1/2 p = 0.2347; 1/3 p = 0.0002; 2/3 p = 0.0098 

7.9489 

0.0008 

1-2 

COMPOSITE INDIV - lED 

y2 p = 0.1386; 1/3 p = 0.0002; 2/3 p = 0.0149 

8.2399 

0.0006 

3-4 

COMPOSITE INDIV-Ambush 

1/2 p = 0.0243; 1/3 p = <0.0001; 2/3 p = 0.0057 

13.4983 

<0.0001 

6-8 

COMPOSITE INDIV-Weapons 

2.2811 

0.1101 

10- 

13 

COMPOSITE INDIV - Communications 

2.2122 

0.1174 


Table 10. Analysis of before and after responses to statements related to 
composite self-assessed individual proficiency by skill set; yellow 
highlight indicates overall significant difference within company, bold 
indicates between-platoon significant difference 

Next, we looked at the skill group composite scores to see if this 
trend held across all skill sets. We used the same analysis methodology of 
starting with ANOVA to determine if there was a difference between platoons and 
then using pairwise t-tests to find which platoons differed. We found a difference 
between platoons for reaction to lED, reaction to ambush, vehicle recovery, and 
casualty evacuation. For reaction to lED, reaction to ambush, and casualty 
evacuation, the trend from the overall composite score was reflected; both 
simulation-trained groups felt that their proficiency increased after training, 
whereas the traditionally trained group did not. For vehicle recovery, only the full 
mission simulation trained group showed a significant improvement in perceived 
proficiency versus the traditionally trained group. For reaction to ambush, all 
three platoons were significantly different from each other; this was the only 
domain in which the two simulation-trained platoons assessed their proficiency 
differently. The full mission simulation trained platoon’s self-assessed proficiency 
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increased more than that of the part-task simulation trained platoon. For 
weapons employment and communications, there was no significant different 
between platoons in self-assessed tactical proficiency. 

We examined more closely the tactical skill groups that showed 
significant differences by platoon to see if the individual scores reflected the 
trends of the parent composite grouping. Table 11 lists the results. For reaction 
to lED, this was the case. Both questions showed a significant difference 
between platoons, with the simulation-trained platoons having a higher perceived 
proficiency after training than the traditionally trained platoon. The reaction to 
ambush questions differed, however. Both questions showed a difference 
between platoons. For reaction to blocked ambush, the full mission simulation 
platoon had higher self-assessed proficiency than the part-task simulation trained 
platoon; the part-task simulation trained platoon did not differ from the 
traditionally trained platoon. Data analysis for the reaction to the unblocked 
ambush question followed the trend of the overall composite score, in which both 
simulation-trained platoons showed significance for higher self-assessed 
proficiency than the traditionally trained platoon, but did not differ from each 
other. 
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# 

QUESTION 

F RATIO 

P(>F) 

1 

INDIV - React to an unexploded lED 

1/2 p = 0.0947; 1/3 p = 0.0003; 2/3 p = 0.0437 

7.1301 

0.0016 

2 

INDIV - React to an lED detonation 

1/2 p = 0.3034; 1/3 p = 0.0010; 2/3 p = 0.0191 

6.2862 

0.0031 

3 

INDIV - Take action against a blocked ambush 

1/2 p = 0.0189; 1/3 p = <0.0001; 2/3 p = 0.0910 

8.6607 

0.0005 

4 

INDIV - Take action against an unblocked ambush 

1/2 p = 0.1573; 1/3 p = <0.0001; 2/3 p = 0.0017 

11.7701 

<0.0001 

5 

INDIV - Cordon and 360 degree security 

2.7234 

0.0730 

6 

INDIV - Employ vehicle machine guns / weapons 

2.3307 

0.1051 

7 

INDIV - Mounted fire and maneuver 

1.7102 

0.1886 

8 

INDIV - Shift fires / cease fires 

3.0269 

0.0551 

9 

INDIV - Vehicle recovery / bump plan 

1/2 p = 0.4481; 1/3 p = 0.0106; 2/3 p = 0.0677 

3.6691 

0.0308 

10 

INDIV - Casualty evacuation 

1/2 p = 0.1756; 1/3 p = 0.0366; 2/3 p = 0.0008 

6.2918 

0.0031 

11 

INDIV - Communication with higher headquarters 

1/2 p = 0.1469; 1/3 p = 0.0064; 2/3 p = 0.1874 

3.9630 

0.0236 

12 

INDIV - Communication between vehicles in a 
convoy 

0.2459 

0.7827 

13 

INDIV - Communication between personnel in 
vehicle 

0.7653 

0.4692 


Table 11. Analysis of before and after responses to statements related to self- 
assessed individual proficiency by individual question; yellow highlight 
indicates overall significant difference within company, bold indicates 
between-platoon significant difference 

In summary, we saw a difference between platoons in tactical skills 
overall. In general, the simulation-trained platoons did not differ from each other, 
but both were higher than the traditionally trained platoon for self-assessed 
tactical proficiency improvement. Reaction to blocked ambush did not follow this 
trend, because the two simulation-trained platoons differed, with the full mission 
trained platoon having the higher self-assessed proficiency. Casualty evacuation 
did not follow this trend either, because only the full mission simulation-trained 
platoon showed significant increase versus the traditionally trained platoon. The 
weapons and employment and communications skill groups showed no 
difference between platoons. 


118 





















Next, we looked at composite scores, and then each question 
individually, by platoon to determine whether there was a significant difference in 
self-assessed proficiency before and after training. We took the difference in 
Likert scaled responses from each question and conducted a t-test investigating 
the null hypothesis that there was no change in self-assessed proficiency. 
Across all three platoons and all questions, there was no decrease in self- 
assessed proficiency; participants did not feel they got worse regardless of 
platoon or skill domain. For the overall tactical composite score, both of the 
simulation-trained platoons increased in self-assessed proficiency, whereas 
analysis showed no change with the traditionally trained platoon. All platoons 
showed no difference for communication between vehicles in the convoy or 
communication between personnel in the vehicle. The full mission simulation 
trained platoon felt they had improved with significance for all other skill sets; the 
part-task simulation trained platoon agreed for all skills except cordon / 360 
degree security and shift / cease fires. On the other hand, the traditionally 
trained platoon did not show any change in self-assessed proficiency for any 
skill. That is, the Marines in the traditionally trained platoon assessed their 
individual proficiency for all analyzed skills the same regardless of whether they 
had had the traditional training or not. Table 12 summarizes this analysis. 
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Question 

MAP 1 

tTest Prob>|t| 

MAP 2 

^ tTest Prob>|t| 

MAP 3 L 

' tTest Prob>|t| 

Composite 

4.9092 

<0.0001 

3.3374 

0.0030 

-0.36678 

0.7171 

C lED 

7.9530 

<0.0001 

3.2874 

0.0034 

0.1088 

0.9143 

C AMB 

6.3471 

<0.0001 

3.3249 

0.0031 

-0.8614 

0.3979 

C WEA 

2.8395 

0.0095 

1.8433 

0.0788 

0.2413 

0.8115 

CCOM 

2.2170 

0.0373 

0.2277 

0.8220 

-0.4845 

0.6326 

PROF 1 

7.1895 

<0.0001 

2.5171 

0.0196 

0.0000 

1.0000 

PROF 2 

5.3911 

<0.0001 

3.8724 

0.0008 

0.1819 

0.8572 

PROFS 

5.1627 

<0.0001 

2.0765 

0.0497 

-0.6174 

0.5430 

PROF 4 

6.0055 

<0.0001 

3.2137 

0.0040 

-1.0000 

0.3277 

PROFS 

2.5981 

0.0164 

1.5554 

0.1348 

-0.5692 

0.5748 

PROF 6 

3.0262 

0.0062 

3.2137 

0.0040 

0.9010 

0.3769 

PROF 7 

2.1054 

0.0469 

2.4721 

0.0216 

0.0000 

1.0000 

PROFS 

2.4721 

0.0216 

-0.6171 

0.5435 

-0.3271 

0.7466 

PROF 9 

3.4583 

0.0022 

2.7324 

0.0122 

0.2531 

0.8024 

PROF 10 

2.5543 

0.0181 

4.7997 

<0.0001 

-0.5489 

0.5884 

PROF 11 

4.7500 

<0.0001 

2.6470 

0.0147 

0.3852 

0.7036 

PROF 12 

-0.1817 

0.8574 

-1.4168 

0.1705 

-1.2817 

0.2127 

PROF 13 

0.7676 

0.4509 

-0.6802 

0.5035 

-0.8106 

0.4259 


Table 12. Analysis of before and after responses to statements related to self- 
assessed individual proficiency by platoon; yellow highlight indicates 
significant increased proficiency 


c. Unit Tactical Proficiency 

The before and after surveys featured the same tactical skill set for 
unit proficiency as for individual proficiency. Thus, participants used the same 
five-point Likert scale grading system discussed above, but they rated their unit’s 
proficiency for each skill instead of their own. An example demonstrates the 
utility of this evaluation mechanism. A Marine may feel that he is quite proficient 
at reaction to ambush, but his platoon’s proficiency does not match his own 
because of newly joined Marines, unit leadership, or some other reason. 
Because the survey construct matched the individual survey described above, 
we used the same analysis methodology. 

We used the same mechanism of creating composite questions 
and breaking the questions down as necessary depending on significance found. 
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That is, we developed an overall composite tactical skill, composed of lED, 
ambush, vehicle recovery, casualty evacuation, weapons employment, and 
communications composite skills. We used ANOVA to test the null hypothesis 
that there was no difference by platoon in ratings for the overall unit tactical 
composite score. We found evidence to reject this hypothesis with an F-ratio of 
9.5772 and a p-value of 0.0002. Pairwise t-tests by platoon indicated the same 
trend as in the individual composite tactical score; that is, the full mission and 
part-task simulation training platoons did not differ significantly from each other, 
but they both differed from the traditionally trained platoon. In all cases, the 
traditionally trained platoon’s self-assessed unit proficiency increased less than 
that of the simulation-trained platoons, as shown in Table 13. 


# 

QUESTION 

F RATIO 

P(>F) 

All 

COMPOSITE UNIT-Tactical 

1/2 p = 0.5399; 1/3 p = 0.0001; 2/3 p = 0.0012 

9.5772 

0.0002 

1-2 

COMPOSITE UNIT - lED 

1/2 p = 0.3117; 1/3 p = <0.0001; 2/3 p = 0.0004 

12.9869 

<0.0001 

3-4 

COMPOSITE UNIT-Ambush 

1/2 p = 0685; 1/3 p = <0.0001; 2/3 p = 0.0153 

9.9240 

0.0002 

6-8 

COMPOSITE UNIT-Weapons 

1/2 p = 0.6791; 1/3 p = 0.0053; 2/3 p = 0169 

5.1195 

0.0093 

10- 

13 

COMPOSITE UNIT - Communications 

1/2 p = 0.2732; 1/3 p = 0.0070; 2/3 p = 0.1090 

4.0401 

0.0233 


Table 13. Analysis of before and after responses to statements related to 

composite self-assessed unit proficiency by skill set; yellow highlight 
indicates overall significant difference within company, bold indicates 
between-platoon significant difference 

As in the individual case, we investigated the composite skill groups 

within the overall tactical skill set to see if all skill groups followed the same trend. 

We found that the trend held for all composite skill groups, so we examined to 

the final level to determine whether the trend extended to the individual questions 

within each skill group. For lED, ambush, vehicle recovery, and casualty 

evacuation, the trend remained the same. Flowever, the trend did not extend to 

individual questions for two of the tactical skill groups. Within the weapons 

employment skill group, the “Shift fires / cease fires” skill violated the trend. In 
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this case, the full mission and part-task simulation-trained platoons differed from 
each other, but the part-task simulation-trained platoon did not differ from the 
traditionally trained platoon. Within the communications composite group, only 
the “Communications with higher headquarters” skill followed the trend. There 
was no significant difference between platoon for “Communication between 
vehicles in a convoy” and “Communication between personnel in a vehicle.” This 
unit rating trend for the unit communications composite group mirrored the trend 
for the individual communications group. Table 14 shows these results. 


# 

QUESTION 

F RATIO 

P(>F) 

1 

UNIT - React to an unexploded lED 

1/2 p = 0.1531; 1/3 p = <0.0001; 2/3 p = 0.0006 

14.1274 

<0.0001 

2 

UNIT - React to an lED detonation 

1/2 p =0.5686; 1/3 p = 0.0002; 2/3 p = 0.0009 

10.0209 

0.0002 

3 

UNIT - Take action against a blocked ambush 

1/2 p = 0.0502; 1/3 p = 0.0005; 2/3 p = 0.0963 

7.0846 

0.0019 

4 

UNIT - Take action against an unblocked ambush 

1/2 p = 0.1470; 1/3 p = <0.0001; 2/3 p = 0.0033 

11.1145 

<0.0001 

5 

UNIT - Cordon and 360 degree security 

1/2 p = 0.5859; 1/3 p = 0.0050; 2/3 p = 0.0202 

5.0354 

0.0099 

6 

UNIT - Employ vehicle machine guns / weapons 

1/2 p = 0.3144; 1/3 p = 0.0259; 2/3 p = 0.0012 

6.1877 

0.0038 

7 

UNIT - Mounted fire and maneuver 

1/2 p = 0.8513; 1/3 p = 0.0328; 2/3 p = 0.0203 

3.6778 

0.0319 

8 

UNIT - Shift fires / cease fires 

1/2 p = 0.0142; 1/3 p = 0.0033; 2/3 p = 0.6681 

5.2798 

0.0080 

9 

UNIT - Vehicle recovery / bump plan 

1/2 p = 0.9619; 1/3 p = 0.0250; 2/3 p = 0.0259 

3.6621 

0.0323 

10 

UNIT - Casualty evacuation 

1/2 p = 0.2584; 1/3 p = 0.0006; 2/3 p = <0.0001 

13.4421 

<0.0001 

11 

UNIT - Communication with higher headquarters 

1/2 p = 0.8775; 1/3 p = 0.0005; 2/3 p = 0.8775 

9.2914 

0.0003 

12 

UNIT - Communication between vehicles in a convoy 

1.6562 

0.2004 

13 

UNIT - Communication between personnel in vehicle 

1.4421 

0.2454 


Table 14. Analysis of before and after responses to statements related to self- 

assessed unit proficiency by individual question; yellow highlight indicates 
overall significant difference within company, bold indicates between- 
platoon significant difference 
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Thus, the unit tactical skill ratings showed similar trends to the 
individual skill ratings. There was a significant difference in overall composite 
score. The difference occurred between the simulation-trained platoons and the 
traditionally trained platoon but not between the two simulation-trained platoons. 
This trend extended through most of the composite skill groups and individual 
skills. Notable exceptions included weapons employment and communications. 
For communications, the trend in differences only extended to communication 
with higher headquarters but not to internal unit communications. 

As in the individual case, we examined each question individually 
by platoon to determine whether there was a difference in self-assessed unit 
proficiency before and after training. We used the same methodology of 
conducting t-tests on the differences between the responses to test the null 
hypothesis that there was no change in self-assessed unit proficiency. Here, it 
became readily apparent that the results did not match the trends of the 
individual analysis; the overall tactical composite rating demonstrated this, as 
shown in Table 15. Overall, neither simulation-trained platoon changed in self- 
assessed unit proficiency. On the other hand, the traditionally trained platoon 
showed a decrease in self-assessed unit proficiency with an F ratio of -3.1907 
and p-value of 0.0041. That is, the simulation-trained platoons felt that their 
unit’s proficiency was the same regardless of the training whereas the 
traditionally trained platoon rated their unit’s proficiency worse after training. For 
the full mission simulation trained platoon, self-assessed unit proficiency only 
increased for the lED and ambush skills; there was no change for all other skills. 
The part-task simulation-trained platoon showed an increase in self-assessed 
unit proficiency for one lED skill, employing vehicle machine guns, and casualty 
evacuation. They showed a decrease in self-assessed unit proficiency for shift 
fires / cease fires internal communications between vehicles and personnel. The 
part-task simulation-trained platoon showed no change for all other skills. The 
traditionally-trained platoon showed a decrease in self-assessed unit proficiency 
in all skills except reaction to blocked ambush and vehicle recovery. 
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Table 15. Analysis of before and after responses to statements related to self- 

assessed unit proficiency by platoon; yellow highlight indicates significant 
increased proficiency, red highlight indicates significant decreased 

proficiency 


d. Simulation Group Qualitative Ratings 

Participants in both simulation groups rated nine statements about 
the training on the survey taken after training. We designed these statements to 
gain insight into the experience of simulation training itself. We used one way 
ANOVA to test the null hypothesis that there was no difference between platoons 
on the responses to these statements. We failed to reject the null hypothesis. 
We analyzed the responses to determine whether any individual statements 
yielded surprising results. Since we found no difference between platoons, we 
conducted this analysis in the aggregate by combining the responses from both 
groups. In general, participants responded to all questions with average 
markings near the middle rating. The two statements that a satisfied participant 
would rate lower indeed showed lower responses than the other seven 
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questions. In summary, analysis of the qualitative ratings showed no difference 
between platoons and produced no interesting information worthy of report. 
Figure 11 shows these results. 




This training mission was successful 






During this exercise, I felt like my actions in the 
virtual environment had no consequences 

During this exercise, I felt like I was playing a game 








During this exercise, I felt like I was conducting 
training 

During this exercise, I felt like I was part of a group 
working together 

During this exercise, I felt isolated from others 


< 




< 





This computer simulation provided sufficient audio cues 
for me to know what was going on 

This computer simulation provided sufficient visual cues 
for me to know what was going on 

The training value of this exercise came from the 
debriefing and not the exercise itself 


1 1.5 2 2.5 3 3.5 4 4.5 5 


Figure 11. Analysis of qualitative ratings of simulation training experience 


e. Comments 

Participants had ample opportunity to provide qualitative feedback 
on the last page of the surveys taken after the training. Several questions served 
to guide the participants’ comments. We asked participants to list the easiest 
and most difficult task for each of individual, vehicle crew, and unit performance. 
We asked what the participant liked least and most and then provided space for 
free comments. We examined this feedback by grouping the feedback 
comments into categories within each feedback question. For example, some 
participants responded that situational awareness was most difficult for them as 
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an individual, so we grouped these responses together. While this informal 
methodology provided no means for statistical analysis, it did allow us to examine 
the multitude of purely qualitative comments for trends. 


Also, we conducted an open interview session with each simulation 
training platoon at the end of the training. We used a standard set of questions, 
which included the following: 

• What did you like about the training? 

• What did you not like about the training? 

• Did you like the progression of the scenarios? 

• Would you rate the training with a thumb up or a thumb 
down? 

When we combined the organized qualitative input from the survey 
forms with the group comments in the interview, we noticed several points worth 
mention. For the simulation-trained platoons. Marines commented on 
communications more than any other topic area, and the comments were both 
positive and negative. Throughout all of the simulation training, communications 
posed the most technical glitches, so this problem received ample critique. Most 
Marines had an opinion about how participants should communicate as a vehicle 
crew, but their opinions did not necessarily agree. Some liked using the voice 
communications as a crew because it was more like the real vehicle. Others 
would have preferred to use the computer communications with a headset like a 
vehicle intercom. Since the vehicle commander had to wear a headset to use 
the convoy communications channel, he could not hear voice communications as 
well as the radio communications, contributing to this issue. On the other hand, 
convoy communications received many positive comments. Many felt that the 
simulation exercise had helped their unit improve in the area of unit 
communications. They wanted the driver to be able to hear the convoy radio net 
like in a real HMMWV. In general, trainees liked the challenge and training 
experience of the convoy radio net, but did not like the vehicle crew 
communication or technical glitches imposed by JVTR. 
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Both simulation groups remarked about the challenges of 
situational awareness. Both groups felt that the simulation training helped their 
unit in this area, but the training also posed some artificial challenges. For 
example, audible feedback such as explosions and gun shots come from the 
computer speakers, so they provide no directional cues like in the real world. On 
the battlefield, reactions to explosions occur instinctually because a person can 
determine relative direction and distance, but a laptop computer provides 
inadequate cueing for this. Also, VBS 2™ does not provide a view out of the 
driver’s window, so the driver cannot see on the left side of the vehicle. The 
vehicle commander’s view is very limited on the right side of the vehicle. The 
result is that the vehicle commander and driver have much less visibility than in 
the real world and must depend on the gunner to know what can be seen from 
the vehicle’s point of view. 

Both simulation platoons spent much time in the interview sessions 
discussing the keyboard interface. In short, the majority of the participants 
sought a console game-type controller for the simulation instead of a keyboard 
interface. They felt that such a controller would be more intuitive and more 
familiar from other leisure gaming activities. 

In general, simulation-trained participants liked the progression of 
the scenarios. The full mission training platoon felt that the progression of 
scenarios started and ended at appropriate levels. The part-task training 
platoon, on the other hand, felt as a group that the early scenarios were too easy. 
However, the convoy commander adamantly disagreed and pointed out that the 
easy scenarios were very useful to him to work out the bugs in his immediate drill 
execution before facing the complications of the more advanced scenarios. Both 
simulation platoons viewed the Twentynine Palms terrain as lower fidelity and 
useful for introductory training while the Sahrani terrain was considered higher 
fidelity and more useful for advanced scenarios. Figures 12 and 13 show scenes 
from the two terrain databases. 
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Both simulation groups commented extensively on the AAR tool. 
Both groups believed the AAR tool was critical in providing a big picture view of 
what happened during the exercise. They liked the playback capability and liked 
seeing where others had been in the main action events of the training. Most 
participants felt like they had very little understanding of the enemy activity until 
seeing the AAR material, and they felt the AAR tool clarified the enemy situation 
considerably. Both groups felt the AAR tool was one of the most valuable 
aspects of VBS 2™ training. 

Both simulation platoons reported that the training helped to 
develop the unit’s immediate action drills. Immediate action drills, particularly 
reaction to ambush, were often listed as easy tasks for the simulation platoons. 
Both platoons often listed immediate action drills as one of the primary benefits of 
the simulation training. 

The traditional training platoon’s feedback tended to focus more on 
the conduct of immediate action drills than the feedback of the simulation 
platoons. The traditional training platoon rarely mentioned situation awareness 
or unit communication in their feedback. Most notable in the traditional training 
platoon’s feedback was the lack of trend at all. Most Marines responded to the 
easiest and hardest questions with some aspect of an immediate action drill, but 
the responses were not particularly consistent in any way. The single point of 
consistency in the feedback was that the training took a long time for what was 
accomplished. 

One common point among all three training groups was the desire 

to train on Mine Resistant Ambush Protected (MRAP) vehicles. At the time of the 

training, all MRAP vehicles were deployed to combat theater, and thus none 

were available for training. VBS 2™ does not contain MRAP vehicles as it is 

deployed on the DVTE suite. However, the Joint Training Counter-Improvised 

Explosive Device Operations Integration Center (JTCOIC) has used VBS 2™ as 

a platform for its activities and has developed an MRAP vehicle that can be 

loaded easily onto any VBS 2™ platform. Given the impending combat 
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deployment, the Marines’ desire to train with MRAP vehicles was 
understandable, and with available modification, VBS 2™ could have satisfied 
the need. Currently, the Marine Corps does not allow such local modification to 
DVTE suites, relying instead on centrally provided updates, so the MRAP from 
JTCOIC could not be used in this training. 

Likewise, we conducted an open interview session with the platoon 
commanders who served as trainers for the three groups. The platoon 
commander who conducted the traditional training had used VBS 2™ in the past, 
so he had a perspective of the potential of the simulation. All three officers 
agreed that the strength of the simulation lay in the AAR tool’s ability to aid the 
trainer in providing feedback to the training audience. Still, all three officers 
demonstrated skepticism about using personal computer-based training. They 
were not convinced that the training was worth the technical troubles of setting it 
up. Furthermore, the simulation platoon commanders felt that the view available 
in a personal computer was too restrictive for infantry training. All three officers 
agreed that training on a personal computer would pay off on deployment at sea, 
where no other training was possible and time was readily available to set up the 
system. 


f. Live Exercise 

The company participated in a six day live field exercise with two 
primary training objectives: urban patrolling and mounted patrolling (Figure 14). 
Both exercises were based out of the Kilo 2 combat town area in Camp 
Pendleton. The mounted patrolling training package contained two segments: 
an initial short course and a longer, more comprehensive course. Marines ran 
the exercises at the platoon level, and vehicle constraints resulted in each 
platoon maneuvering with six vehicles. As previously discussed, the short 
convoy course met our needs as an evaluation mechanism for the simulation and 
traditional preparatory training. This course consisted of a roughly three-mile 
loop with three major events. During the exercise. Marines were exposed to an 
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unexploded lED to gauge reaction expertise. They established and conducted a 
hasty vehicle checkpoint. The final station was initiated by a simulated lED 
detonation, and then a two-man oppositional force unit ambushed the convoy. 



Figure 14. Marines operating vehicle check point in live convoy exercise 

A cadre of six Marines controlled the entire operation, including the 
urban patrolling. A colonel served as the exercise controller and officer in charge 
and, for the most part, remained at a base station near the combat town. The 
controller team had essentially no military resources; they relied on personal 
vehicles or rides in the participating unit’s vehicles for transportation and they 
relied on the participating unit’s communications assets. Three Marines shared 
responsibility for evaluating the mounted patrolling courses, but all three Marines 
had other responsibilities as well. Typically, one or two Marines would monitor 
the short convoy course. For all short-course evolutions, a single evaluator rode 
in one of the participating platoon’s vehicles. This unit evaluator dismounted 
whenever the unit stopped and monitored the action from whatever best vantage 
point could be gained on the ground. During movement, the unit evaluator 
simply rode in the vehicle. The controller cadre had the option to augment the 
unit evaluator with a second Marine using overwatch positions, or perches, as 
shown in Figure 15. For each of the three stations, the terrain provided a 
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suitable perch to view the participating platoon’s activity from an overhead view. 
Using the unit’s Motorola radios, the controller cadre had sufficient 
communication to control the exercise and interact to provide meaningful insight 
into what was happening with the participating platoon on the ground. 



Figure 15. Watching a convoy movement from an overwatch position 

In previous exercises, the controller cadre used a standardized, 
custom-made form to record observations and provide debriefing feedback to the 
training unit. The form divided the exercise into its primary tasks and subtasks 
and provided lines to write information about each. In order to collect data, we 
modified the existing form by adding squares for each of ten vehicles. The 
controllers used a three-point scale to grade each vehicle on each of the tasks 
and subtasks. The scale included named markings of “untrained,” “trained,” and 
“mastery.” Evaluators placed one of the three letters U, T, or M in the boxes to 
grade each vehicle for each subtask. If the exercise was evaluated by more than 
one Marine, the unit controller riding in one of the training unit’s vehicles served 
as the primary controller and recorded all grades. The augmenting controller or 
controllers provided feedback to the primary, who incorporated the information 
into his overall grade. 
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Each of the platoons conducted two fully evaluated runs of the 
short convoy course. With virtually no exception, the evaluators rated the 
platoons as “trained” for each of the tasks and subtasks for all participating 
vehicle crews. The overwhelming number of “trained” ratings resulted in no 
significant difference by platoon, by vehicle crew, or by task category. 
Observational comments from controller cadre indicated agreement that there 
was no noticeable difference in platoon performance on the short convoy course. 

g. Notes about Experiment Execution 

(1) Controller Effects. For the simulation training platoons, 
we gradually and deliberately increased the involvement of the platoon 
commanders who were acting as trainers. We recognized the importance of the 
trainer as the link between the unit’s training objectives and the simulation 
exercise. We built all the scenarios used in the training before the training was 
conducted. With this done, trainers could involve themselves with the simulation 
in two ways: alter the scenario during the exercise run using the real-time editor, 
and run the after-action tool to display action from the exercise that best 
supported the learning points they wished to convey. 

The concept of “building scenarios” can have different 
meanings for different trainers and training systems. In the context of this 
project, a “built scenario” included all players and gear in an initial starting 
position, all enemy in initial starting positions, and moderate amounts of objects 
that amplified the scene. Examples of amplifying objects included civilians along 
the street, civilian vehicles parked in driveways, and dirt pile cues near lED sites. 
Even a well-built scenario does not run by itself. The controller must monitor the 
enemy to ensure their actions are plausible, and advanced scenarios included 
traffic and moving civilians that had to be added and controlled through 
waypoints and other techniques. Thus, the real-time editor in VBS 2™ provided 
ample opportunity for the trainer or controller to affect the exercise to meet 
training objectives. As the controller, we did all this work without the assistance 
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of the trainer. Throughout the course of the week, we gradually involved the 
trainer more until the trainer was doing most of the facilitation in the final 
exercises. 

Likewise, the after-action tool records the exercise and 
allows a user to re-play the exercise while operating a camera that can view the 
exercise from any angle. Both the real-time editor and the after-action tool allow 
the user to view exercise activity from a two-dimensional map view with icon 
representations of players, or from a three-dimensional world view with avatar 
representations. Between the two views, the freedom of movement throughout 
the exercise environment, and the ability to re-play segments multiple times, the 
user has many different options available for using the after-action tool to make a 
learning point clear. It is conceivable that two different trainers could have 
completely different debriefs based on the same scenario by supporting different 
learning points with various camera angles and world views. 

As we increased the involvement of the two platoon 
commanders, it became apparent that their styles of involvement differed. The 
full mission simulation training platoon commander used both the real-time editor 
and after-action tool aggressively to drive home the learning points he 
considered most important. At times, we had to reign in his real-time editor 
involvement in order to keep the exercise standardized enough to maintain the 
integrity of the experiment. The full mission platoon commander readily took 
charge of the after-action tool, manipulating the camera on his own to drive home 
his points about unit performance. On the other hand, the part-task simulation¬ 
training platoon commander took a much more reserved approach to both tools, 
relying more on the controller for support. The part-task platoon commander was 
noticeably more reluctant to assume control of the after-action tool, allowing the 
controller to choose the playback world view and camera position and then 
tailoring his comments based on what was projected. We did not anticipate this 
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sort of difference between trainers, so we did not build a mechanism into the 
experiment to quantify it. Nevertheless, the difference was noticeable throughout 
the exercise week. 

(2) Communications. We sought to run the experiment 
using VBS 2™ as Marines would use it straight out of the boxes of the DVTE 
suite with no other external support. With this in mind, we relied exclusively on 
the Joint Virtual Tactical Radio (JVTR) for communications. JVTR is a Single 
Channel Ground and Airborne Radio System (SINCGARS) radio simulator used 
to train Marines how to use the SINCGARS radio. This simulator can be used as 
a communications platform for other simulations in DVTE. In DVTE, one 
facilitates communication with VBS 2™ on a single computer by running JVTR 
simultaneously. This setup works well enough, but one must understand that the 
communications platform is an external add-on and not an internal component of 
VBS 2™. In this exercise, we only had a single radio net. The vehicle 
commanders used the net to communicate as a convoy unit. The convoy 
commander also used this net to communicate with the platoon commander 
acting as trainer. In this capacity, the trainer served as a higher headquarters to 
receive battlefield reporting and provide headquarters guidance. The controller 
also used this net to provide administrative instructions, as necessary, such as 
exercise start and end information. 

We had many technical problems with communications, 
some of which interfered with the exercise execution. JVTR simulates a dual 
radio set with one radio going to one earpiece of the headset and the second 
radio sounding in the other earpiece. Since we were only using one net, we used 
only one radio. During the first two days, all of the JVTR radios would stop 
functioning at some point in the day. Through trial and error, we determined that 
we could fix the problem by programming both JVTR radios with the same net 
frequency. Once VBS 2™ was launched, we could not change the radios without 
stopping the simulation. This created a situation where it was usually better to 
have the participant continue the exercise without communications rather than 
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shut the simulation down, fix the radio, and then try to get the participant back 
into the play of the exercise. We got around this problem by operating VBS 2™ 
in Windowed mode. Eventually, we ran all JVTR radios with the same frequency 
and the problem went away. However, the continuous communications troubles 
in the first two days caused a discernable amount of distress in the conduct of 
the exercise. 

We also found that the JVTR radios would periodically shut 
down for the busiest users. Quite simply, a busy JVTR radio saturated the 
computer’s buffer until the application locked up. We noticed that this problem 
occurred mostly with the convoy commander, so we solved the problem by 
providing communications for him on a separate computer. The convoy 
commanders readily adapted to the slight inconvenience of operating with two 
laptops. We drastically reduced JVTR problems in this way, and the few 
problems that did occur for the convoy commander could be remedied quickly. 

(3) Technical SNAFUs (Situation Normal: All Fouled Up). 
We sought to use the scripting in VBS 2™ to facilitate the casualty evacuation 
and vehicle recovery training. A trigger allows scripting on the fly to accomplish 
various simple tasks. For both skills, we wanted to have explicit control of which 
individual or vehicle was a casualty. We used a fake lED to give the effect of an 
explosion. A fake lED produces an explosion but has no associated damaging 
effect. We then used a trigger to disable a certain vehicle or cause a certain 
casualty. Originally, we had these triggers pre-built in the scenario. We used the 
scripting reserved word “this” in the trigger. Unfortunately, this caused the effect 
to be applied to the whole unit instead of the single entity breaking the trigger 
bound. For example, we set a trigger to injure the driver of the vehicle that 
tripped the trigger, but all of the drivers in the convoy ended up getting injured at 
once. After a couple exercises, we fixed this problem. Since the part-task 
simulation-training platoon did casualty evacuation and vehicle recovery in the 
first two days, this problem affected them exclusively. While most participants 
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understood the technical problem and appreciated it as a humorous mistake, the 
part-task trained platoon got an unfair sampling of technical glitches up front 
because of this error. 

In previous studies, we had noticed a computer glitch in 
which the vehicle crew representations got mixed up in the simulation. In a 
vehicle crew, a gunner would be unable to control his avatar and would 
continuously get thrown out of the vehicle despite repeated attempts to get back 
in. The vehicle’s driver would get thrown out of the vehicle and be unable to get 
back in. The driver still would be able to drive the vehicle using keyboard 
controls. However, the driver would be left standing on the ground while the 
vehicle drove away, based on his keyboard input, such that the driver could not 
see where the vehicle was going. This glitch happened so rarely that we could 
not figure out how to fix it or replicate it for diagnosis. Marines often have the 
initiative to work through even the most inconvenient of problems, and a few 
vehicle crews managed to continue with the driver using the gunner’s screen to 
control the vehicle and the gunner repeatedly getting back in the vehicle every 
few seconds. We eventually found that we could quite easily remedy this 
situation by changing either the driver’s or the gunner’s computer to map view 
and then pressing “Esc” to get back into the world view. However, through our 
initial lack of understanding of the problem—and their gallant initiative—a few 
Marines suffered undue frustration because of this problem. 

(4) OPORD (Operational Order) Delivery. We intended to 
run the simulation training with minimal input to the unit’s training implementation. 
We wanted to remain facilitators while the unit leadership remained trainers, and 
we sought to avoid blurring that line, if possible. We provided the basic 
scenarios and map materials in support of the training, but we did not provide 
operational orders that specified the background and mission of the exercise. 
We relied on the platoon commanders and their staff for this function. To keep 
the workload minimal, we provided an OPORD shell, included in Appendix R, to 
help their development efforts. 
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We noticed less input from the platoon staffs in the area of 
OPORD development and delivery than we had intended. We had tried to allow 
the platoon to use the blank slate of the scenario at hand to best suit unit needs, 
but instead ended up with a rather blank slate for the participants. The Marines 
relied heavily on the OPORD shell we provided, reading from it almost verbatim. 
Participants did not add in detail from the maps and terrain to tailor to the 
exercise at hand. Moreover, they did not add in their own Standing Operating 
Procedures (SOPs) and custom guidance. Thus, participants operated 
throughout the exercise based mostly on the provided map. Little planning 
affected the conduct of the exercise. However, the full mission simulation¬ 
training platoon put noticeably more thought and effort into OPORD preparation 
and delivery. While the simulation training we conducted certainly did not orient 
on mission planning skills, we could detect the differences in platoon preparation 
levels in exercise execution. 

D. DISCUSSION 

1. Hypotheses 

At the most general level, this study allowed us to answer the two base 
questions quite simply. Preliminary training with VBS 2™ positively impacts 
performance in a platoon-level tactical scenario, so we reject the first null 
hypothesis. Specifically, the simulation training resulted in no difference in 
academic performance, significant difference in self-assessed proficiency, and no 
difference in live training performance. Additionally, there was not a difference 
between using a full mission approach versus a part-task approach to training 
when using VBS 2™ for platoon-level tactical training, so we fail to reject the 
second null hypothesis. This conclusion held across all three evaluation 
mechanisms. 

We must interpret the results associated with the first hypothesis carefully. 
Essentially, Marines felt better about their tactical skill but failed to demonstrate 
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improvement in a live setting. One must keep in mind the confounding factors of 
the human element in both the participation and evaluation of this exercise. With 
this in mind, one can use this study to conclude that the simulation training did no 
harm. That is, the simulation training was at least as good as the traditional 
training. However, a conclusion that these results prove that simulation training 
is better than traditional training would, in our opinion, risk extrapolating the 
information too far. 

One must also remember that self-assessed proficiency did not differ 
significantly for all skill sets. Most notably, the training seemed to impact 
communications less than other skill sets, particularly communications at the 
vehicle crew level. At the individual level, the training did not impact self- 
assessed proficiency for weapons skills. It is also important to note that not all 
significant differences occurred as the result of improvement. For self-assessed 
unit proficiency, virtually all skills showed a significant difference, but the 
difference occurred because the control group’s self-assessed proficiency 
decreased after training. This result may not suggest as much about the virtual 
training as it does about the traditional training. It may be that the traditional 
training revealed unit training deficiencies that never got corrected during the 
exercise. 

Further, one must be aware of the limitations of the study. Between the 
two simulation groups, the potential for a trainer effect was demonstrated through 
the different techniques of using the real-time editor and after-action review tool. 
Technical glitches in both the communication and the simulation application itself 
interfered with the training experience. Finally, the evaluation of the live exercise 
was not robust enough to highlight fine differences in performance. While these 
limitations do not make the study results useless, they do impact the extent to 
which the results can be generalized. 
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2. Insights 


We would like to differentiate this study from previous pilot studies 
discussed in this thesis. The previous pilot studies involved groups of people 
who were not, in any way, representative of the unit structure involved in the 
training. This study, on the other hand, involved an infantry company with a unit 
structure established specifically for motorized patrolling and convoy operations. 
While it is inherently obvious that this difference would change the training 
experience, it is worthy of mention. The simulation training worked far better with 
an intact leadership structure. Participants understood who was in charge, basic 
standard procedures, and individual personalities of the unit. These differences 
changed the face of the training completely. For the type of convoy training 
studied in these experiments, unit integrity is a key factor. Putting individuals 
through VBS 2™ training does not necessarily mean a unit or the individuals 
themselves are trained for the operation at hand. Rather, the training effect, if 
there is to be any at all, comes from training the unit as it will train live and as it 
will fight in combat. 

This project also highlighted a difference in the training audience between 
the reserve infantry company and the active duty training cadre at Weapons and 
Field Training Battalion. We asked for a show of hands for both of the simulation 
platoons if the Marines were college students immediately prior to mobilization. 
Only a few Marines had not been enrolled full-time in college immediately prior to 
deployment to Camp Pendleton. We did not work to determine the courses 
these Marines had been taking or their academic goals and interests. 
Nevertheless, the mere fact of their almost-universal college enrollment 
separates these reservists from their active duty counterparts to some degree. 
The assumption that the findings from this study can be generalized to all 
Marines is confounded, to some extent, by this factor. 

Some are quick to tout personal computer-based simulation as a training 
solution in order to appease the technophilic generation of young adults who fill 
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the junior enlisted ranks of the Marine Corps. While these young Marines 
typically owned computers and used cell phones, they were not necessarily 
gamers. Moreover, the infantry is often a target audience for personal computer- 
based simulation, and the infantryman may be less technophilic than his 
counterparts in other job specialties. While both groups assessed themselves as 
more proficient in the selected tactical skills, neither group appreciated simulation 
training more after the exercise. Moreover, the verbal interview after the exercise 
resulted in mixed reactions to the training. Some Marines liked the training, 
some did not, and most had somewhat ambivalent views toward it. Readily 
assuming that a Marine will appreciate simulation training simply because he 
uses a personal data device and the World Wide Web is a fallacy. Marines will 
appreciate training that makes them better warfighters regardless of the level of 
technology involved. 

Both platoon commanders and participants lauded the after-action review 
tool as critical to the positive VBS 2™ experience. Likewise, proactive facilitation 
of the exercise impacts training. Unit leadership can be involved in both the run¬ 
time execution of the exercise and the debriefing afterward. Further, VBS 2™ 
can accommodate different leadership styles effectively. However, using the 
real-time editor and after-action review tools effectively involves a learning curve. 
One can easily overcome this with a few exercise runs, but one cannot assume 
that unit leaders will get the most effect out of the simulation training based only 
on their tactical expertise. Leaders must get their hands dirty with the simulation 
operation. 

As we increased the level of platoon commander involvement in the use of 
the real-time editor and after-action review tools, we noticed subtle differences in 
their use. Typically, the platoon commanders used the third-person world view 
during the conduct of the exercise. Administratively, we typically used the map 
view. During the debriefing, platoon commanders started the week with a 
preference for the third-person world view and started using the map view more 
as the week went on. The views differ in the level of detail provided to the eye. 
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The third-person world view makes the scene look realistic, but the map view 
shows important battle information, such as unit location and weapon orientation, 
explicitly. We found that most debriefing comments that were supported by the 
after-action review tool tended to involve one of three topics: firing unit’s sight 
perspective, weapon orientation and shot direction, and entity position relative to 
other entities. The third-person world view is more suitable to see a situation 
from a gunner’s point of view. The map view better demonstrates the direction of 
a shot; in fact, this information is explicitly displayed as a line on the map. 
Relative position is often more easily viewed at the macro level with the map 
view, but in close terrain with hills or buildings, the third-person world view is 
more useful. One could study Best Practices for the use of the different tools at 
length, but it is worthy of note here that the best use requires practice and 
experience. Just as a vehicle driver must learn the simulation interface to drive 
in the trainer as in the real world, the trainer must learn to use the real-time editor 
and after-action tools to train effectively. 

Scenario testing is critical to a positive training experience. An exercise 
controller can easily test many aspects of the scenario by himself while 
constructing the scenario. VBS 2™ provides a very convenient way to test run 
the scenario from the editor. However, this practice does not test the scenario 
with group interaction. We missed the technical problem of all drivers becoming 
casualties as the result of one trigger because we were unable to test the 
scenario with a group. A single developer cannot make a finished training 
product; the developer must be able to test his exercise on a group of people. 
Fortunately, a controller on his toes can correct most mistakes in stride with the 
real-time editor, but a unit with critical dependence on a given scenario should 
ensure that it has been group tested before execution. 

One must put much thought into exercise control. We designed all 
exercise components with a single controller in mind for the practical reasons of 
the experiment. We relied heavily on artificial intelligence driven enemy, 
civilians, and traffic. For the most part, we made all of these entities work 
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effectively for the given training problem. With the single controller in mind, we 
put much effort into scripting these entities in scenarios beforehand. We found in 
practice that it was easier in most cases to create entities in stride using the real¬ 
time editor. This is not to say that a controller should start with a completely 
blank screen; the decision to add entities before exercise versus during the 
execution is a balancing act predicated on controller experience and ability. A 
more complex exercise could easily overwhelm a single controller. One must 
make consideration for multiple controllers and support for players acting as 
civilians and oppositional forces. 

JVTR is a workable solution for exercise communications, but it may not 
be the best solution. JVTR, even operating at its best, brings the possibility of 
radio crashes due to buffer overload. At its worst, JVTR introduces 
communication problems that cannot be fixed during exercise execution. The 
problems are more difficult to diagnose than any VBS 2™ issues. By the very 
nature of communications, JVTR problems frustrate users and undermine the 
effectiveness of the simulation as a situational-awareness training tool. 
Certainly, a better solution would be an internal communications capability in 
VBS 2™ itself. In the absence of such software, a unit might be better advised to 
use external communications systems. This consideration is noteworthy when 
considering the out-of-the-box Best Practices for DVTE users. 

3. Summary 

Based on the results of this study, we conclude that preparatory training 
with VBS 2™ is at least as good as preparatory training by traditional methods 
such as sand table or garrison rehearsal. We conclude that the simulation can 
be used as a part-task trainer or a full mission training device. With these 
thoughts in mind, personal computer-based training is not the single answer to a 
training problem, but rather another capable and worthy tool for the trainer’s 
toolbox. VBS 2™ does what it was designed to do; that is, it provides the trainer 
flexibility for resource constrained environments. Additionally, personal 
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computer-based training complements traditional training well. When Marines 
tire of the sand table and parade deck, the computer provides an invitingly 
different perspective and experience. Thus, the simulation provides flexibility to 
the trainer. Considered by the dollar, this advantage may not seem worthwhile, 
but when the real world equipment is in transit or otherwise unavailable, the 
training areas are booked, and only a few days remain to get ready, personal 
computer-based training can be an attractive asset. 
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V. EXPERIMENT IN SUPPORT OF HYPOTHESIS 2 


A. INTRODUCTION 

In Experiment 4, we found that training with VBS 2™ produced a positive 
result for small unit tactical convoy training, although changes in performance 
were limited to increases in self-assessed proficiency. With knowledge that 
personal computer-based training was beneficial, we sought some insight into 
how this benefit can best be achieved. We based our work on careful 
contemplation of the trainer effect we had noticed in Experiment 4. That is, the 
full mission platoon commander engaged himself more fully in the conduct and 
debriefing of the simulation exercise than the part-task platoon commander did. 
We noticed this difference despite the distinct effort to involve both Marines. This 
led to the question of how the simulation training experience would differ if the 
direct involvement of the trainer were limited rather than encouraged. 

We felt it important to investigate this effect because of the unique 
potential of the DVTE concept. Quite commonly. Marines searching for VBS 2™ 
will find it set up semi-permanently or permanently in local simulation centers. 
Our personal observation confirms this is true across major Marine Corps 
installations such as MCAGCC in Twentynine Palms, California and the I MEF 
Simulation Center in Camp Pendleton, California. Units have even pooled their 
DVTE resources to create simulation centers such as the Marine Regiment 
simulation center in Kaneohe Bay, Hawaii. In these simulation environments. 
Marine unit trainers implement the training exercise through the support of a 
contractor staff who facilitates the training. Using this construct, the Marine 
trainer provides the contractor with training objectives and specific desires such 
as terrain or equipment. The contractor then develops the appropriate scenario 
and controls the scenario execution for the unit. 
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However, DVTE is different. DVTE puts the personal computer-based 
simulation in the hands of the unit. With this construct, there is no contractor 
support staff. Instead, units send representatives, typically at the junior enlisted 
level, to a two-week course to learn how to operate DVTE as a suite and use 
each of the suite's simulation applications. The DVTE course runs on a regular 
basis at both Camp Lejeune and Camp Pendleton with additional courses at 
other installations such as Twentynine Palms. Thus, the Marine Corps makes 
DVTE training available to all units, and as such, a unit must send some 
representatives to DVTE training before their DVTE suite is fielded. These 
DVTE-trained Marines can then fill the role of the contractor support in the 
Marine Corps' various simulation centers. That is, given a set of training 
objectives and exercise specifications, the Marines can develop training 
scenarios and operate them in support of the unit's training goals. 

Regardless of the perceived complexity of virtual environments, a serious 
game is nothing more than a training device, a tool of the trainer. With this in 
mind, a serious look at training effectiveness is more than an investigation of the 
tool; rather, it involves investigation of the trainer's use of the tool. The trainer's 
use of the tool involves his or her knowledge of the simulation as well as what he 
or she does with it. A simpler example illustrates this point. A hammer is a 
useful tool, and with it, one can build an entire building. However, without 
knowing how to use the hammer, one cannot build anything and may cause 
destruction and frustration in the process of trying. The effectiveness of a 
serious game involves a relationship between the simulation and the trainer. 

In either of the DVTE usage situations described above, a third party 
potentially separates the trainer from direct interaction with the simulation 
exercise. We sought to investigate the potential of VBS 2™ if this separation 
was removed. We have discussed the concept of a Marine using VBS as an 
extension of his warfighting skills, and this idea was central to Experiment 4. In 
Experiment 5, we investigated the concept of a Marine using VBS 2™ as an 
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extension of his training skills. We questioned whether the trainer's knowledge of 
the simulation could impact the simulation experience. 

This chapter summarizes Experiment 5, in which we tested the following 
hypothesis: 

• A trainer who has been trained in the operation of the simulation 
and is thereby able to control the simulation exercise will produce 
more effective training than a trainer who effects the simulation 
exercise through a third party. 

B. METHOD 

We turned to a school environment to test our hypothesis. The school 
environment provided a tested structure with an instructor cadre in place. Such a 
situation resulted in rigidly defined trainer roles that reduced the potential for a 
confounding trainer effect in the experiment. We selected the Enlisted 
Professional Military Education (EPME) program at Kaneohe Bay, Hawaii for our 
work. EPME Kaneohe Bay operates a seven-week professional training 
curriculum commonly known as the Sergeant's Course. This course focuses on 
the junior enlisted leader who fits the profile of the trainee at the center of our 
previous convoy training work, so the products we had developed in support of 
the first thesis hypothesis naturally fit in the EPME environment. The Sergeant's 
Course offers the student an overview of a broad array of topics spanning the 
entire Marine Corps from close order drill to equal opportunity. Tactical training 
only receives the brief overview perspective of any other topic area with most 
attention paid to mission planning and execution. Class sizes usually vary 
between twenty and thirty Marines. The instructional package follows a very 
structured and formalized standard managed by the headquarters at Training 
and Education Command (TECOM) in Quantico, Virginia. 

EPME Kaneohe Bay suited our experiment well because the unit had not 
yet benefited from its parent organization's dissemination of VBS 2™ training 
packages from the headquarters in Quantico. The school had a full DVTE suite 
on site but had not yet opened it for any purpose other than basic inventory. 
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EPME Kaneohe Bay was ready to start training with simulation, though. On the 
school's own initiative, the instructor cadre used white space in the course 
schedule to conduct a convoy simulation exercise at the Virtual Combat Convoy 
Trainer (VCCT). For a half or full day of each course, depending on schedule 
constraints, EPME students would run standard convoy exercises intended to 
familiarize them with the execution of a tactical plan. Since students hailed from 
any given job specialty in the Marine Corps, the school did not expect tactical 
prowess and was not particularly concerned with learning any given tactical skill. 
Rather, the trainer focused on the troop leading procedures, planning steps, and 
decision-making skills of a tactical operation and used the convoy exercise 
execution as a forum for the students to practice these concepts in an actual 
mission. 

1. Participants 

Participants in the experiment included three Sergeant’s Course classes 
at EPME Hawaii. The experiment involved seventy-nine participants altogether 
with twenty-four participants in the first class, twenty-three participants in the 
second class, and thirty-two participants in the third class. All participants were 
Marine Corps Sergeants. We did not keep demographic statistics for the first 
class. For the second and third classes, the Sergeants had an average of 5.81 
years in the Marine Corps (standard deviation 1.72 years). Participants 
averaged 25.33 years in age (standard deviation 3.11 years). Participants came 
from every facet of the Marine Corps with fifty different job specialties 
represented, including Marines from ground, aviation, and logistics elements. 
Eleven Marines participated in the study as trainers, although only five Marines 
saw the study from beginning to end due to personnel turnover. Trainers ranged 
in rank from Corporal to First Sergeant, averaged 28.50 years in age (standard 
deviation 4.81 years) and 8.98 years in service (standard deviation 4.44 years). 
Seven of the trainers had infantry backgrounds; the others came from supporting 
job specialties throughout the Marine Corps. 
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For the simulation-trained groups, we collected data about computer and 
simulation experience, convoy experience, and use of simulations versus sand 
tables in exercise planning and rehearsal. We used ANOVA or chi-squared tests 
as appropriate to test the null hypothesis that the groups did not differ any of the 
categories. We found no evidence to reject the null hypothesis for any 
demographic category. 

2. Apparatus 

VCCT provides a full mock-up of military vehicles surrounded by large 
screen projections to provide a fully immersive training experience. The VCCT at 
Kaneohe Bay contains six large bays, each with an individual replica of a 
HMMWV or MTVR. Trainees can participate in the exercise seated in the 
various seats of the vehicle, the gunner's station, or the floor area around the 
vehicle. In the case of the MTVR, participants can sit in the cargo area of the 
vehicle as well. Large screens around the bay, combined with a robust sound 
system, provide fully immersive visual and audio cues. Typical scenarios involve 
skill sets examined in our earlier work, including reaction to lED, reaction to 
ambush, casualty evacuation, and vehicle recovery. A contractor staff operates 
the simulation facility with most activity occurring in a control room, providing a 
standard exercise overview from an array of computer screens. The unit trainer 
occupies this room, providing exercise input and tracking execution details with 
the help of the contractor support. 

As in previous experiments, we accomplished VBS 2™ training using the 
DVTE suite. In this case, we used the freshly unpacked equipment at EPME 
Hawaii. Specifically, the experiment used eighteen Dell Precision M6300 laptops 
that were networked using D-LINK DGS-2208 8 port switches. Peripheral 
equipment, including mice, cables, and switches, came from the standard DVTE 
package. The network configuration was the same as for all training exercises, 
although the instructor cadre dismantled the equipment and re-packed it between 
different segments of the training. We networked all computers together using 
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switches, allowing all machines to participate together in the same environment. 
As shown in Figure 16, we used the forty-seat classroom for the simulation 
training with trainees occupying the first two rows of sixteen seats each. 


Figure 16. EPME instructors training a Sergeant’s Course class with VBS 2™ 

at Kaneohe Bay 

Unlike previous experiments, we used the Combat Net Radio (CNR) in 
combination with the internal VBS 2™ communications. CNR is a proprietary 
software package designed to enable communications for a wide variety of 
simulations. Caltryx, the company who developed CNR, has cooperated with 
Bohemia Interactive to develop a CNR interface that operates with the VBS 2™ 
AAR tool. CNR has two components: CNR SIM serves as the radio 
communications, and CNR LOG serves as the recorder. CNR SIM allows users 
to select multiple nets, but CNR LOG records all radio traffic on the network 

regardless of net. For this reason, we only used CNR for convoy net 
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communications. For communications within the vehicle, we used the internal 
VBS 2™ vehicle communications feature. This feature allows players to 
communicate depending on their vehicle association in the scenario. 

3. Procedure 

We designed our experiment to compare three treatments. In the first 
treatment, trainees received no simulation training at all. This control group 
served as a baseline. In the second treatment, students trained in the VCCT with 
the instructor cadre administering the exercise and conducting after action 
reviews through the third party support of the simulation facility contractors. In 
the third treatment, we trained the instructor cadre in the operation of VBS 2™. 
The instructors then used VBS 2™ on their own, with no contractor support, to 
conduct the training exercise. For convenience, we refer to these treatment 
groups as the control group, VCCT group, and VBS 2™ group, respectively. 
Because of the small class size, we followed three separate EPME classes, each 
class serving as an individual group. The experimental groups went in the order 
listed above; that is, the first class was the control group, the second was the 
VCCT group, and the final class was the VBS 2™ group. 

We used two evaluation mechanisms to determine the effectiveness of the 
training for each group. First, we developed a survey mechanism based on our 
products for Experiments 2, 3, and 4. Participants filled out a demographic 
survey on which they rated their self-assessed skill proficiency and attitude 
toward simulation. They completed a post training survey with similar self- 
assessed skill ratings and simulation attitude ratings. They also answered some 
questions about their simulation training experience. The Experiment 5 surveys 
differed from the previous surveys in the addition of skills pertaining to troop 
leading procedures, mission planning, and decision making. We included the 
demographic survey as Appendix T and the post training survey as Appendix V. 
Second, we used a knowledge test. We used the knowledge tests developed for 
Experiments 3 and 4 as a basis for the Experiment 5 tests. We also used the 
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same paired test design discussed in the previous experiments. That is, we used 
two different tests, each with twenty-four multiple choice questions. The 
questions were paired between tests. For example, if one test featured a 
question about reaction to lED, the other test featured a different question with 
similar intent. Like the surveys, the Experiment 5 tests differed from our previous 
versions because they contained questions related to troop leading procedures, 
mission planning, and decision making to reflect the goals of the EPME training. 
We included the two knowledge tests as Appendices Y and Z. Since the control 
group did no simulation training, they only took the knowledge tests as a 
baseline, answering all forty-eight test questions in one composite test. The 
VCCT and VBS 2™ groups took the survey and knowledge test before and after 
simulation training. 

For the VBS 2™ group, we developed a training package for the EPME 
instructor cadre so that they could administer simulation training to the EPME 
class. We sought to provide a turn-key set of tools that the trainers could use at 
their discretion to effect the course's training objectives. We based the package 
on previously developed convoy scenario products. Specifically, we created a 
package that began with the Familiarity training exercise developed in 
Experiment 1. The convoy training involved a modified version of the full mission 
training package used in Experiment 4, starting with the Training Wheels 
scenario to get participants working together as a convoy team and then 
progressing to the Noble Pass and Rainbow Canyon scenarios in the Twentynine 
Palms terrain and the Ixel and Iguana scenarios in the Sahrani terrain. In 
addition to the scenarios themselves, the training package included the hot key 
cheat sheet developed in Experiments 1 and 2, all maps and the OPORD shell to 
support the scenarios, and the slide show we used to support the Familiarity 
training in all previous experiments. In general, we gave EPME our products 
from the Experiment 4 full mission training. 

We conducted the instructor cadre training during a down period between 
Sergeant's Course classes over a three-day period (Figure 17). Ten instructors 
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participated in the training, although ongoing commitments caused a few of the 
Marines to miss portions of the training. We started the first day of training by 
conducting the Familiarity exercise. We did not run the exercise exactly like 
previous experiments, because we spent extra time teaching the Marines how to 
run the Familiarity training by themselves. We followed the Familiarity training 
with an introduction to the communications systems and then had the Marine 
staff complete the Training Wheels scenario as a single convoy unit. Again, we 
focused on showing the Marines how to use the scenario themselves as the 
trainer. We spent the latter half of the first day teaching the Marines how to build 
a scenario by going through the full menu of VBS 2™ entities and explaining 
each. The entire training exercise was practical application with the Marine staff 
doing each trained task at his or her own laptop following our example shown by 
screen projection. On the second day, we introduced the Marines to the AAR 
tool, showing them how to switch views, find the action, and use the tool to meet 
training goals. For the remainder of the second day, we went through each of 
the four convoy scenarios in detail. Marines took turns acting as the controller 
while the rest of the staff participated in the exercise. We provided individual 
instruction to the Marine acting as controller throughout the exercise, focusing on 
the use of the real time editor to use triggers to achieve battlefield effects, 
provide the experience of civilians in the environment, and manipulate computer 
controlled enemy to achieve training goals. We projected the controller's screen 
on the Proxima so that the rest of the class could see what the controller did as 
we introduced new real time editor concepts. Thus, the second day of training 
was a combination of practical application, practice as a user, and individualized 
instruction oriented on giving Marines experience as an exercise controller. 
Throughout the training, we followed a list of notes that guided our instruction 
and provided the Marines an outline of the training as well as administrative 
information such as file locations and passwords. We included this notes sheet 
as Appendix V. 
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Figure 17. Instructor training with VBS 2™ at EPME Kaneohe Bay 


We sought to investigate the effectiveness of the train-the-trainer package 
and the instructor cadre's impression of the simulation training. We used a pair 
of surveys. We administered the demographic survey after the train-the-trainer 
instruction. The demographic information was the same as all other surveys 
used in previous experiments. As in the other surveys oriented on simulation 
trainees, we asked the instructors for their attitude toward simulation training. 
We asked the Marines to rate how effective they expected VBS 2™ training to be 
for each of the tactical skills in the trainee surveys. We also asked the Marines 
to rate their proficiency in a variety of VBS 2™ controller skills. Finally, we asked 
for subjective comments about the train-the-trainer instruction. The post training 
survey asked for the same ratings of VBS 2™ effectiveness to train the various 
tactical skills. It also involved the same self-assessed proficiency for VBS 2™ 
controller skills. The post training survey asked questions about the simulation 
experience from the controller point of view. Finally, Marines rated which 
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simulation trainer they would prefer for each of the tactical skills, choosing 
between VCCT and VBS 2™. The survey also provided the opportunity for 
subjective comments. We included the demographic survey as Appendix U and 
the post training survey as Appendix W. As in previous experiments, all survey 
products for Experiment 5 involved questions with five-point Likert scaled 
responses. 

As in previous experiments, we used statistical group comparison 
techniques including the paired t-test for analysis. For all statistical tests, we 
used an alpha level of 0.05. 

C. RESULTS 

The study involved multiple measures of effectiveness. The knowledge 
tests administered before and after the virtual training provided insight into 
whether participants’ factual knowledge of the skill areas changed as a result of 
training. Participant surveys administered before and after training measured 
changes in the individuals’ self-assessed proficiency, providing a subjective look 
at changes in tactical performance as a result of the training. Comments on the 
participant surveys provided further subjective insight into the value of the 
training. Finally, instructor surveys administered before and after the training 
provided subjective information from the trainer’s viewpoint. We will discuss the 
results of each of these evaluation mechanisms in turn and conclude with some 
notes about the study overall. In short, none of the evaluation mechanisms 
produced significant results in the data analysis. Flowever, observation of the 
exercise in progress combined with analysis of the subjective comments 
provided noteworthy insights about the impact of the trainer on simulation 
training. 

1. Knowledge Tests 

This study involved three groups: the control group who received no 
virtual training; the VCCT group in which trainers implemented training through a 
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third party; and the VBS 2™ group in which trainers implemented training directly 
by administering the simulation training themselves. The control group took a 
knowledge test with no virtual training at all in order to establish a baseline. The 
two treatment groups took knowledge tests before and after their virtual training. 
We graded all of the multiple choice tests and recorded the results in a 
spreadsheet for analysis, paying particular attention to the average number of 
questions missed overall. 

We started by comparing the percentage of questions missed for each 
individual by treatment group. We used ANOVA to test whether there was a 
difference in performance on the final test by group, and we found no significant 
difference (F ratio of 0.2194, p-value of 0.8035). We also investigated whether 
the knowledge of either simulation trained group changed after training, using a t- 
test. For both groups, no significant difference in the number of correct answers 
resulted (for the VCCT group, t Ratio of 0.3184 and p-value of 0.7517; for the 
VBS 2™ group, t Ratio of -0.6644 and p-value of 0.5089). 

The control group’s knowledge test consisted of fifty multiple choice 
questions. We dropped two questions and split the test in half to make two tests 
of twenty-four questions each for the two treatment groups. The VCCT group 
took the two tests as planned with half taking Test A first and the other half taking 
Test B first. However, an administrative error resulted in the VBS 2™ group 
taking the fifty-question test instead of Test A. That is, the VBS 2™ group either 
took Test B or the entire fifty-question test (which included the twenty-four 
questions from Test B). We corrected this problem in the data analysis by only 
grading those questions that were on Test A. However, participants from the 
VBS 2™ group who took the fifty-question test before training saw the Test B 
questions twice. 
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2. Participant Surveys 


We used the same techniques to analyze the participant self-assessed 
proficiency that we used in Experiment 4. Like the previous experiment, we 
analyzed individual attitude toward simulation, individual tactical proficiency, unit 
tactical proficiency, simulation group qualitative ratings, and comments. In 
general, responses from the VCCT group reflected a positive training experience 
and improved self-assessed proficiency while the VBS 2™ group’s responses 
reflected much less deviation from the status quo. In total, we analyzed 
responses from twenty-three participants in the VCCT group and thirty-two 
participants in the VBS 2™ group. 

a. Individual Attitude toward Simulation 

As in Experiment 4, we compared responses to five-point Likert 
scaled questions from surveys taken before and after the VCCT and VBS 
training. We started by creating a composite score of the four simulation attitude 
questions by averaging the answers. We used a t-test to test the null hypothesis 
that the platoons did not differ in their change in attitude toward simulation. With 
an F-ratio of 4.1614 and a p-value of 0.0465, we rejected the null hypothesis and 
concluded that the two treatments differed with the VCCT group having a higher 
assessed attitude than the VBS 2™ group. We scrutinized the individual 
questions to see if this difference was reflected across all questions. The trend 
held for all questions except “I think a unit should use computer-based simulation 
in its tactical training.” For this item, the responses reflected no significant 
difference. Table 16 shows this analysis. 
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# 

QUESTION 

F-RATIO 

P(>F) 


COMPOSITE - Attitude toward simulation 

16.8145 

0.0001 

1 

Computer-based simulation is an effective training 
tool. 

4.1614 

0.0465 

2 

Today's planned training will improve my ability to 
conduct convoy operations. 

17.3745 

0.0001 

3 

Today’s planned training will improve my unit's ability 
to conduct convoy operations. 

23.5620 

<0.0001 

4 

1 think a unit should use computer-based simulation 
in its tactical training. 

3.1431 

0.0822 


Table 16. Analysis of before and after responses to statements related to attitude 
toward personal computer-based training; yellow highlight indicates 
significant difference in attitude between VCCT and VBS 2™ groups 


Next, we analyzed the questions by group to see how each 
treatment group’s attitude toward simulation changed as a result of the training 
(Table 17). After calculating the differences between before and after responses, 
we used a t-test to test the hypothesis that the group’s attitude toward simulation 
did not change. For the VCCT group, we found evidence to reject the null 
hypothesis and determined that the group’s cumulative attitude toward simulation 
improved. Individual scrutiny of each question reflected this trend with the 
exception of “Today’s planned training will improve my ability to conduct convoy 
operations.” On the other hand, while the VBS 2™ group’s cumulative attitude 
toward simulation reflected no significant difference, the two questions 
specifically to “today’s training” showed a significant decline in attitude. 



Table 17. Analysis of before and after responses to statements related to attitude 
toward personal computer-based training by virtual training group; 
yellow highlight indicates significant increased attitude, red highlight 
indicates significant decreased attitude 
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In summary, the attitudes of the two simulation groups differed. 
Both groups showed an increased inclination to believe that a unit should use 
simulation in its tactical training. However, when evaluating the impact of 
“today’s training,” the VCCT group believed more strongly after the training that 
the training helped at both individual and unit levels while the VBS 2™ group 
rated these questions lower. Additionally, the VCCT group’s belief that 
simulation is an effective training tool increased after the training while the VBS 
2™ group’s opinion remained the same. 

b. Individual Tactical Proficiency 

As in Experiment 4, participants rated their proficiency in various 
skills before and after training, and we compared the responses to determine if 
the training positively or negatively impacted their opinion of their skills. As 
shown in Table 18, we started with composite scores and broke down the scores 
to the individual question level. Analysis began with ANOVA to test the null 
hypothesis that the changes in self-assessed proficiency did not differ between 
the two simulation groups. The overall composite score for all skills provided 
evidence to reject the null hypothesis and conclude that the groups differed. 
Specifically, the VCCT group’s self-assessed proficiency increased more than 
that of the VBS 2™ group. Breaking the analysis down by category and then to 
the individual skill level demonstrated that this trend extended through all skills 
with the exception of “Shift fires / cease fires.” 
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# 

QUESTION 

F RATIO 

P(>F) 

All 

COMPOSITE INDIV - Tactical 

25.2534 

<0.0001 

I 


COMPOSITE INDIV - lED 

13.3330 

0.0006 

1 

INDIV - React to an unexploded Improvised 
Explosive Device 

14.7006 

0.0003 

2 

INDIV - React to an Improvised Explosive Device 
detonation 

11.2612 

0.0015 

1 


COMPOSITE INDIV-Ambush 

7.4722 

0.0085 

3 

INDIV - Take immediate action against a blocked 
ambush 

8.9945 

0.0041 

4 

INDIV - Take immediate action against an unblocked 
ambush 

4.1814 

0.0459 

1 

5 

INDIV - Cordon and 360 degree security 

8.8151 

0.0045 

1 


COMPOSITE INDIV - Weapons 

10.4512 

0.0022 

6 

INDIV - Employ vehicle machine guns / weapons 

14.3834 

0.0004 

7 

INDIV - Mounted fire and maneuver 

8.4603 

0.0053 

8 

INDIV - Shift fires / cease fires 

3.5529 

0.0651 

1 

9 

INDIV - Vehicle recovery / bump plan 

9.9786 

0.0027 

1 

10 

INDIV - Casualty evacuation 

11.8349 

0.0012 

1 


COMPOSITE INDIV - Communications 

34.6655 

<0.0001 

11 

INDIV - Communication with higher headquarters 

22.6119 

<0.0001 

12 

INDIV - Communication between vehicles in a 
convoy 

24.7351 

<0.0001 

13 

INDIV - Communication between personnel in 
vehicle 

26.2611 

<0.0001 

1 


COMPOSITE INDIV - Preparations 

19.9135 

<0.0001 

14 

INDIV - Execute the troop leading steps using 
BAMCIS 

10.1581 

0.0024 

15 

INDIV - Conduct mission analysis using METT-T 

15.1204 

0.0003 

16 

INDIV - Receive a 5 paragraph operational order 

8.8146 

0.0045 

17 

INDIV - Give a 5 paragraph operational order 

16.0316 

0.0002 


Table 18. Analysis of before and after responses to statements related to self- 
assessed individual proficiency; yellow highlight indicates significant 
difference between simulation training groups with VCCT group having the 

higher ratings 
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Next, we calculated the difference between before and after self- 
assessed proficiency ratings for each individual and conducted t-tests by group 
to test the null hypothesis that each group’s proficiency rating did not change 
(Table 19). For the VCCT group, we found evidence to reject the null hypothesis 
with self-assessed proficiency showing significant increase for all categories and 
individual skills. On the other hand, for the VBS 2™ group, only two categories 
showed significant difference: both the ambush and weapons skills 

demonstrated increase in self-assessed proficiency. Ratings showed no 
significant difference for other categories and skill sets. 


Question 

August (VCCT) 
tTest L P>'ob>|t| 

November {VBS 2) 
tTest L Pi'ob>|t| 

1 CALL 

5.6930 

<0.0001 

1.6593 

0.1075 

1 C lED 

4.6351 

0.0001 

1.4076 

0.1695 

1 PRO 1 

5.0091 

<0.0001 

1.2223 

0.2311 

1 PRO 2 

4.2040 

0.0004 

1.5323 

0.1359 

1 GAME 

4.8465 

<0.0001 

3.2621 

0.0028 

1 PROS 

4.4770 

0.0002 

2.5593 

0.0158 

1 PRO 4 

4.5917 

0.0001 

3.3211 

0.0024 

1 PROS 

4.5344 

0.0002 

0.3485 

0.7299 

1 C WEA 

5.3531 

<0.0001 

2.6955 

0.0114 

1 PROS 

6.0694 

<0.0001 

1.0441 

0.3048 

1 PRO 7 

5.6000 

<0.0001 

2.8304 

0.0082 

1 PROS 

3.9056 

0.0008 

2.6833 

0.0117 

1 PRO 9 

4.8062 

<0.0001 

1.5626 

0.1286 

1 PRO 10 

4.6986 

0.0001 

1.0946 

0.2824 

1 CCOM 

5.503471 

<0.0001 

-1.3374 

0.1912 

1 PRO 11 

5.3735 

<0.0001 

-1.4384 

0.1607 

1 PRO 12 

4.9698 

<0.0001 

-0.6485 

0.5216 

1 PRO 13 

5.0070 

<0.0001 

-0.9411 

0.3542 

1 C PRE 

5.0952 

<0.0001 

0.2314 

0.8186 

1 PRO 14 

3.2188 

0.0040 

-0.2543 

0.8010 

1 PRO 15 

3.7607 

0.0011 

-0.8915 

0.3798 

1 PRO 16 

3.8935 

0.0008 

1.2223 

0.2311 

1 PRO 17 

5.5235 

<0.0001 

0.7695 

0.4476 


Table 19. Analysis of before and after responses to statements related to self- 
assessed individual proficiency by simulation group; yellow highlight 
indicates significant increased proficiency 
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In summary, we observed a difference between changes in self- 
assessed proficiency. Regardless of skill domain, the VCCT group’s self- 
assessed proficiency increased. However, the VBS 2™ group’s proficiency 
ratings only changed for two skill groups. We noted no decrease in self- 
assessed proficiency ratings for either group or for any skill category. 

c. Unit Tactical Proficiency 

Just as participants rated individual skill sets before and after virtual 
training, they assessed unit proficiency for the skill sets as well. We used the 
same methodology to analyze the results, starting with ANOVA to test the null 
hypothesis that changes in self-assessed proficiency were the same by treatment 
group. As shown in Table 20, we found evidence to reject the null hypothesis in 
the case of the composite item and all subcategories and individual skills (F-ratio 
25.4943, p-value less than 0.0001). Using a t-test to investigate the null 
hypothesis that each group did not experience a change in self-assessed 
proficiency, we found that the self-assessed proficiency ratings for all unit 
composite categories and individual skills increased for the VCCT group, as 
shown in Table 21. On the other hand, self-assessed proficiency for the VBS 2™ 
group decreased overall. Vehicle recovery, casualty evacuation, and 
communication accounted for this decrease; self-assessed proficiency ratings 
had no change for the other skill sets for the VBS 2™ group. 
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# 

QUESTION 

F RATIO 

P(>F) 

All 

COMPOSITE UNIT - Tactical 

27.4943 

<0.0001 

I 


COMPOSITE UNIT-IED 

23.2832 

<0.0001 

1 

UNIT - React to an unexploded Improvised Explosive 
Device 

20.0585 

<0.0001 

2 

UNIT - React to an Improvised Explosive Device 
detonation 

25.7603 

<0.0001 

1 


COMPOSITE UNIT - Ambush 

15.8090 

0.0002 

3 

UNIT - Take immediate action against a blocked 
ambush 

17.8025 

0.0001 

4 

UNIT - Take immediate action against an unblocked 
ambush 

13.0047 

0.0007 

1 

5 

UNIT - Cordon and 360 degree security 

19.1706 

<0.0001 

1 


COMPOSITE UNIT - Weapons 

22.9595 

<0.0001 

6 

UNIT - Employ vehicle machine guns / weapons 

18.8279 

<0.0001 

7 

UNIT - Mounted fire and maneuver 

21.2568 

<0.0001 

8 

UNIT - Shift fires / cease fires 

23.8085 

<0.0001 

1 

9 

UNIT - Vehicle recovery / bump plan 

27.5881 

<0.0001 

1 

10 

UNIT - Casualty evacuation 

21.4048 

<0.0001 

1 1 


COMPOSITE UNIT -Communications 

32.0918 

<0.0001 

11 

UNIT - Communication with higher headquarters 

31.1784 

<0.0001 

12 

UNIT - Communication between vehicles in a convoy 

29.8752 

<0.0001 

13 

UNIT - Communication between personnel in vehicle 

30.2392 

<0.0001 


Table 20. Analysis of before and after responses to statements related to self- 

assessed unit proficiency by individual question; yellow highlight indicates 
significant difference between simulation training groups with VCCT group 

having the higher ratings 
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Question 

August (VCCT) 
tTest Prob>|t| 

November {VBS 2) 
tTest Prob>|t| 

U C ALL 

4.6770 

0.0002 

-2.3976 

0.0229 

U C lED 

4.8492 

0.0001 

-1.3713 

0.1804 

U PRO 1 

4.4853 

0.0003 

-1.3128 

0.1992 

U PRO 2 

5.1514 

<0.0001 

-1.3926 

0.1740 

U C AMB 

4.1309 

0.0006 

-0.9939 

0.3282 

U PROS 

4.0105 

0.0008 

-1.3932 

0.1738 

U PRO 4 

4.1576 

0.0006 

-0.5835 

0.5639 

U PROS 

5.3785 

<0.0001 

-1.1791 

0.2476 

U C WEA 

5.2778 

<0.0001 

-1.8082 

0.0806 

U PROS 

5.2050 

<0.0001 

-1.4850 

0.1480 

U PRO 7 

5.0000 

<0.0001 

-1.6473 

0.1099 

U PROS 

4.6599 

0.0002 

-2.1818 1 

0.0371 

U PRO 9 

4.9242 

<0.0001 

-2.5668 

0.0155 

U PRO10 

4.3507 

0.0003 

-2.1356 

0.0410 

U CCOM 

4.3261 

0.0004 

-3.5634 

0.0012 

U PR011 

4.0559 

0.0007 

-3.8545 

0.0006 

U PR012 

4.3529 

0.0003 

-3.2IIM 

■■30 

U PR013 

4.3333 

0.0004 

-3.2iii^| 

i0027 


Table 21. Analysis of before and after responses to statements related to self- 
assessed unit proficiency by simulation group; yellow highlight indicates 
significant increased proficiency, red highlight indicates significant 

decreased proficiency 


d. Simulation Group Qualitative Ratings 

As in Experiment 4, the participants rated statements about the 
simulation training experience on a five-point Likert scale. We used ANOVA to 
investigate the null hypothesis that the two simulation groups did not differ in their 
responses to these statements. For five of the statements, the VCCT group’s 
responses were significantly higher than those of the VBS 2™ group. Table 22 
highlights these responses. 
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# 

QUESTION 

F-RATIO 

P(>F) 

1 

This training mission was successful 

31.0227 

<0.0001 

2 

During this exercise, 1 felt like my actions in the virtual 
environment had no consequences 

1.3791 

0.2458 

3 

During this exercise, 1 felt like 1 was playing a game 

0.7526 

0.3898 

4 

During this exercise, 1 felt like 1 was conducting 
training 

10.4479 

0.0022 

5 

During this exercise, 1 felt like 1 was part of a group 
working together 

10.9551 

0.0017 

6 

During this exercise, 1 felt isolated from others 

2.5204 

0.1187 

7 

This computer simulation provided sufficient audio 
cues for me to know what was going on 

4.0546 

0.0494 

8 

This computer simulation provided sufficient visual 
cues for me to know what was going on 

1.7890 

0.1871 

9 

The training value of this exercise came from the 
debriefing and not the exercise itself 

20.3209 

<0.0001 


Table 22. Analysis of before and after responses to statements related to the 
overall simulation training experience; yellow highlight indicates 
significant difference between simulation training groups with VCCT 
group having the higher ratings 


e. Comments 

Participants provided free flowing subjective response to the 
training through open-ended questions on the last page of the survey. These 
questions asked what skills were easiest and most difficult for the individual, 
crew, and unit. Additionally, participants responded about how the exercise 
helped them and how it wasted their time. Finally, participants had the 
opportunity to provide any other comments they felt were important. 

Given the statistically proven differences in attitude toward 

simulation and self-assessed proficiency between the simulation groups, the 

comments section demonstrated several common themes between the two 

groups. Communications, including crew communications, unit communications, 

and communications to higher headquarters, topped the list of concerns for both 

groups. Marines from both groups expressed concern that down time between 

scenarios wasted time. Both groups found situational awareness challenging, 
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while they considered driving and shooting very easy skills in the simulation 
environment. Marines from both groups felt that the training helped to provide an 
overview of convoy operations, and they felt that more of the training was 
necessary and appropriate. 

Comments differed in that many VBS 2™ participants provided 
negative overall comments such as “It [VBS 2™ virtual training] did not help the 
unit” and “patience for others learning controls” was a difficult individual skill. 
Other negative comments included “too little time to learn the program,” “no one 
knew what was going on,” and “get trained administrators.” The theme of too 
little training time was very common throughout the VBS 2™ training group with 
twelve participants annotating the comment somewhere in the survey. The trend 
of these comments did not criticize the simulation itself, but rather the 
administration of the exercise as exemplified by the following comment: “it [VBS 
2™] could be a useful tool”. Participants directed some of the criticism toward 
themselves, such as “student discipline was poor.” 

3. Instructor Surveys 

The instructor cadre filled out surveys immediately after their VBS 2™ 
training and then again after conducting the VBS 2™ exercise with the Marine 
class. However, only five instructors filled out the post-exercise survey, and 
because of administrative problems with the surveys, we could only match two 
instructors’ pre- and post-exercise surveys. For this reason, we did not analyze 
most of the data from the surveys. However, it is worth noting that instructors 
rated their own proficiency for twelve administrator skills immediately after their 
training. All eleven instructors rated all skills with a three or higher on a five-point 
Likert scale. 

One set of data, shown in Table 23, is worthy to note, even though it only 
reflects the input of the five instructors who completed the post-exercise survey. 
On one section of the post-exercise survey, the instructors rated whether they 
preferred VCCT or VBS 2™ for each of the tactical skills evaluated in the 
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experiment. The survey used a five-point Likert scale to capture strength of 
preference with one indicating a strong preference for VBS 2™, three indicating 
no preference between the two, five indicating a strong preference for VCCT, and 
two and four indicating moderate preferences. We averaged the ratings for each 
individual skill so that averages below three indicated an overall preference for 
VBS 2™ and averages higher than three indicated a preference for VCCT. Most 
of the ratings were within 0.5 of the “no preference” mark of three. However, a 
pattern emerged in the grouping of the averages. For tactical action skills such 
as weapon employment and reaction drills, instructors preferred VCCT. On the 
other hand, instructors preferred VBS 2™ for communications and tactical 
planning skills. Skills that stood out with strong preferences included a 
preference for VBS 2™ (average 2.0) for casualty evacuation and a preference 


for VCCT for mounted fire and maneuver (4.0). 


# 

QUESTION 

AVERAGE 

ST DEV 

1 

React to an unexploded Improved Explosive 

Device 

3.5 

1.4 

2 

React to an Improvised Explosive Device 
detonation 

3.5 

1.4 

3 

Take immediate action against a blocked ambush 

3.5 

1.4 

4 

Take immediate action against an unblocked 
ambush 

3.5 

1.4 

5 

Cordon and 360 degree security 

2.4 

1.7 

6 

Employ vehicle machine guns / weapons 

3.5 

1.5 

7 

Mounted fire and maneuver 

4.0 

0.9 

8 

Shift fires / cease fires 

3.5 

1.5 

9 

Vehicle recovery / bump plan 

3.0 

1.7 

10 

Casualty evacuation 

2.0 

1.1 

11 

Communication with higher headquarters 

2.7 

1.6 

12 

Communication between vehicles in convoy 

2.7 

1.6 

13 

Communication between personnel in vehicle 

2.5 

1.5 

14 

Execute the troop leading steps using BAMCIS 

2.8 

1.6 

15 

Conduct mission analysis using METT-T 

2.8 

1.6 

16 

Receive a 5 paragraph operational order 

2.8 

1.6 

17 

Give a 5 paragraph operational order 

2.8 

1.6 


Table 23. Analysis of instructor preferences for VCCT or VBS 2™; numbers lower 
than three with yellow highlight indicate a preference for VBS 2™ and 
numbers higher than 3.0 with green highlight indicate a preference for 

VCCT 
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Instructors also had the opportunity to provide subjective comments at the 
end of their training and at the end of the exercise. Comments at the end of the 
instructor training tended to be positive and optimistic, reflecting a confidence in 
the ability to run an exercise. The only negative trend was a desire for more 
training time with VBS 2™. Comments after the exercise execution reflected a 
positive attitude about the training potential of the simulation. However, 
instructors expressed concern about the technical aspects of the training with 
comments such as “not user friendly,” “too many moving parts,” and “time 
consuming setup.” 

4. Notes about Experiment Execution 

Some notes about the exercise execution are helpful in order to properly 
understand both the VCCT and VBS 2™ portions of the experiment. In this 
experiment, our role was primarily observational leaving little opportunity to 
change the course of the exercise. For this reason, the exercises did not always 
progress in the best interests of the study, but the deviations from the desired 
protocol proved instructive in themselves, providing insight into the impact of the 
trainer on exercise effectiveness. 

For the VCCT exercise, the training cadre planned two runs of a convoy 
scenario with Marines changing roles for the second run. The staff had made 
some effort to organize the students prior to the event with a designated convoy 
commander and assistant convoy commander ready in the briefing room at the 
beginning of the exercise. The exercise proceeded with no difficulties from an 
administrative point of view. At the end of the exercise, the training cadre 
conducted a 20-minute after action review session using the VCCT playback 
capability administered by the contractor staff. The contractors spent a half hour 
re-setting the scenario while the Marines prepared for the second run. However, 
the contractors could not get the second run of the scenario to work properly, so 
the second run was aborted and the training ended about an hour earlier than 
anticipated. 
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Many shortfalls in the administration of the training hampered the VBS 2™ 
exercise. The instructor cadre had set up the computers early in the morning of 
the training day. The computers were properly networked and ready from a 
hardware standpoint, but they had just been booted. Only one instructor 
supported the training exercise. When the students were ready to train, 
communications had not been set up and there was no plan to do so. The 
scenario was not loaded. Without even a convoy commander identified, the 
student class was not organized in any way to conduct the training. The staff 
had planned for two hours of VBS 2™ training, which left a very short time to 
familiarize Marines with the interface, introduce a scenario, organize the 
students, conduct the training, and debrief the exercise. Quite simply, the time 
allotted was too short. Interface training was insufficient. Marines received no 
meaningful brief, the scenario was rushed, and the instructor conducted no after 
action review afterward. The instructor attempted two scenarios, but both were 
convoluted by gross situational awareness problems that were exacerbated by 
Marines yelling across the room in the absence of a communications system. 
The instructor typically focused on controlling enemy and neutral entities in the 
VBS 2™ environment, neglecting technical difficulties, higher headquarters 
responsibilities, and evaluation note taking and critique. In summary, the 
exercise was poorly planned, poorly supported, and poorly executed. 

D. DISCUSSION 

1. Hypothesis 

Taking the statistical analysis of this experiment at face value, we find 

evidence to reject the null hypothesis of the experiment. We can conclude that a 

trainer who has been trained in the operation of the simulation, and is thereby 

able to control the simulation exercise, will produce training that is less effective 

to that produced by a trainer who implements the simulation exercise through a 

third party. Specifically, a trainer who implements the simulation exercise 

through a third party will produce better training than a trainer directly using the 
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simulation tool. The improved self-assessment proficiency ratings of the VCCT 
group support this conclusion. Self-assessed proficiency increased for all tactical 
skill sets for the VCCT group but stayed the same or decreased for the VBS 2™ 
group. 

However, this experiment had too many confounding variables to justify 
such a conclusion about the null hypothesis. Both exercises had flaws in their 
execution that confounded analysis. The poor planning and execution 
associated with the VBS 2™ exercise made the data collected virtually useless. 
Statistical analysis of instructor feedback was meaningless because of the small 
number of available surveys. In short, this experiment provided no means to say 
anything definitive about the null hypothesis. 

However, the conduct of the VBS 2™ exercise provided insight into the 
hypothesis. The crux of the issue examined in this experiment was whether 
Marines received better training if the trainers implemented virtual training 
themselves or through a third party. Clearly, conducting training through a third 
party with VCCT provided a better training experience than instructor 
administered training through VBS 2™. It does not take a lot of statistical 
number juggling to understand why we saw such a result in this case. The VBS 
2™ training was poorly planned and executed. Most likely. Marines would prefer 
a smoothly executed exercise more than a disorganized one no matter the forum, 
skill set, or venue. Thus, the question from this experiment that truly bears 
discussion is why the VBS 2™ training was poorly done. 

2. Insights 

a. Hammering without Knowing Where the Naii is 

Virtual training, no matter what computer platform is used or who 
administers it, is nothing more than a tool to accomplish a training objective. In 
this regard, the virtual training platform is like any other tool: the user must know 
how it works, what to use it for, how to fix it when it breaks, what support it 
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needs, etc. Most importantly, the user must know what he or she intends to 
accomplish with the tool. This statement seems so basic that it threatens to 
insult the reader’s intelligence, but its violation caused problems in the execution 
of the VBS 2™ training in this experiment. 

The senior enlisted leader of EPME Kaneohe Bay expressed the 
training goals for the virtual training very clearly. He sought to put the Marines in 
a tactical situation in which they would have to make a plan, execute the plan, 
and make decisions using teamwork and leadership in order to accomplish an 
objective. Unlike Experiment 4, the instructor staff did not specifically seek to 
improve the group’s convoy skills, although it would have been a desirable side 
effect. Rather, the convoy scenario provided a tactical environment known well 
enough to the average Marine Sergeant that students could jump in and start 
making plans and decisions. 

The execution of the VBS 2™ exercise reflected a loss of focus on 
the original objective. Instead, the instructors became focused on getting 
through the technical details of administering the exercise. Success got a very 
different definition, with instructors simply trying to get through all of the wickets 
to create a viable training environment for the Marines. Whether or not the 
Marines got the opportunity to make reasonable decisions got lost in the turmoil 
of making the event happen. Certainly, the focus on student learning was lost 
when instructors completely omitted the debrief, where decision consequences 
would logically be discussed and critiqued. 

On the other hand, the VCCT training did not immediately 
demonstrate symptoms of this loss of focus problem. However, it is also difficult 
to conclude that the focus was clearly there. Rather, the instructors used a 
canned training program that the contractor staff ran routinely for many different 
units. The contractors knew how to run the scenario, had seen most of the 
variations on execution Marines would make, and knew how to run the playback 
tool prompting the instructor to debrief. The appearance of accomplishing the 
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objective of facilitating a decision-making forum was more the result of contractor 
experience than instructor cadre intervention. 

In summary, a trainer who forges ahead with his or her own training 
program must do so with clear objectives in mind. In the case of the VBS 2™ 
training, the trainers had the tool and knew how to use it. However, a carpenter 
who knows how to use a hammer but just beats on the wood accomplishes 
nothing. He must know where the nail is and hit it with each stroke. The same is 
true of trainer administered simulation exercises. 

b. Practice Makes Perfect 

When we trained the EPME Kaneohe Bay instructor staff to use 
VBS 2™, we spent three days on the training following a curriculum similar to the 
VBS 2™ portion of DVTE training administered by the contractors at the I MEF 
Simulation Center in Camp Pendleton. Such a curriculum is adequate to teach 
the basics of building a scenario, using the real-time editor to administer an 
execution run of the scenario, and using the after action review tool to facilitate 
debriefing. However, administering a VBS 2™ exercise to a group of novice 
users requires quite a bit of knowledge. The trainer must be able to apply that 
knowledge on his or her feet. There is no time to consult other people or 
documentation for an answer, and there is no liberty to experiment and figure out 
the answer. Once the exercise starts running, the administrator must know what 
he or she is doing and be on his or her toes at all times to keep the environment 
realistic for the training audience. 

The three days of VBS 2™ training is clearly insufficient to provide 
this level of application knowledge for the average Marine trainer. It is not that 
more training is required, but rather that the trainer must have the opportunity to 
practice on a live training audience. All of the trainers in this experiment 
improved surprisingly quickly and came out of the experience ready to do a much 
better job the next time. Not only would such experience improve the next 
execution, but it would support much better planning as well. The trained but 
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unpracticed VBS 2™ trainer is really not trained at all. Only practical experience 
can give the trainer the wherewithal to facilitate good training with a tool such as 
VBS 2™. 

Nowhere did the issue of practice become more apparent than in 
time allocation. The two-hour period allotted for the VBS 2™ convoy exercise 
was far too short, even for seasoned administrators and users. The instructor 
cadre had not accounted for the time to load scenarios and set up the networks. 
They had no experience with their scenarios to gauge how long an exercise run 
might take. The resulting plan was a recipe for trouble from the beginning 
because every step of the process had to be rushed. With practice, instructors 
would not have attempted such a plan and would have made adjustments to 
provide a better chance for success. 

c. Spreading the Trainer Too Thin 

Three distinct roles are necessary to implement virtual training 
through a simulation platform like VBS 2™. First, someone must be the 
facilitator. In this role, the trainer has tasks such as making sure everyone gets 
networked into the scenario, causing civilian and enemy entities to act in 
accordance with the training plan, and remembering to record the scenario and 
bookmark it for easy review. Second, someone must be the technician. Almost 
certainly, some members of the training audience will experience technical 
difficulties. The problems may occur because of computer platform issues, 
networking problems, or software glitches. The training audience may induce 
problems through lack of experience, failed attempts to correct a minor problem, 
or accidental actions with unintended consequences. Whatever the source, the 
technician must handle technical problems so that “computer-isms” do not 
overshadow the training. Third, someone must be the planner/evaluator. The 
planner/evaluator determines the training objectives and designs or chooses the 
scenario to meet them. In the planner/evaluator role, the trainer acts as one or 
multiple higher headquarters personnel. The planner/evaluator must take notes 
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about the exercise execution. Most importantly, the planner/evaluator must 
observe both the exercise itself as it unfolds in the virtual environment and the 
training audience as they work through the execution. This observation includes 
close scrutiny of the primary communications nets. For the scope of exercises 
covered in this thesis, a single trainer can fulfill all three of these roles capably as 
long as he or she maintains adequate control over the training audience. More 
likely, a training staff of two or more Marines should administer the exercise, and 
the roles can be divided between available personnel. Whether different people 
split the roles or a single individual performs them alone, all three roles must be 
properly handled for an exercise to produce successful training. 

In the case of the VCCT training administered through a third party, 
the trainer is absolved of facilitator and technician duties. While the contractor 
handles these two roles, the trainer is free to focus on planner/evaluator tasks. 
The cost to the trainer is that direct facilitation is no longer possible and 
adjustments must be described to and interpreted by a third party. 

The division of labor problem became painfully obvious in the 
execution of the VBS 2™ training. The trainer focused on the facilitator role to 
the exclusion of both the technician and planner/evaluator roles. We can explain 
this very simply. The facilitator role is fun. Never does an individual have such 
power as when facilitating a virtual exercise. The facilitator is right in the thick of 
the action making things happen every moment of the exercise. Once one learns 
the interface, it is easy to create complex problems for the training audience, and 
the effect is immediate and satisfying. Such a role appeals to the aggressive, 
controlling mindset of the typical Marine leader. Keeping notes about vehicle 
dispersion or figuring out why someone’s mouse is not working simply fall to the 
wayside when artillery is a finger click away and the enemy RPG team can be 
hidden just a little bit better. Nevertheless, no matter the appeal of any particular 
role, all three roles must receive equal emphasis or chaos will eventually begin to 
take over. 
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Thus, the number of trainers who implement VBS 2™ training is 
important. Curiously, in the VCCT exercise where contractors capably filled the 
facilitator and technician roles, the EPME instructor staff maintained at least two, 
and usually three, instructors to implement the training. On the other hand, when 
all of the demands of virtual training implementation rested on the instructor staff 
alone for the VBS 2™ training, the instructor staff allocated only one trainer to the 
exercise. For a novice staff, this personnel level is insufficient. At a minimum, 
one person needs to be tied to the control station computer while someone else 
is available to move about the training floor to help the training audience. 

3. Summary 

We do not believe that Experiment 5, when viewed in total, supports any 
determination about the null hypothesis. Unfortunately, the execution of the VBS 
2™ exercise was too flawed to properly investigate the effects of direct trainer 
facilitation of a virtual training exercise. However, the flaws in the execution 
themselves illustrated the challenges of the trainer staff simultaneously filling the 
facilitator, technician, and planner/evaluator roles of a virtual training exercise. 
Experiment 5 demonstrated the importance of objective-oriented training, 
practice, and time and support personnel allocation on virtual training directly 
administered by the trainer. Future work could shed light on the potential of 
training in which these challenges are overcome. However, training administered 
through a third party circumvents these problems to a great extent, thereby 
reducing the risk of a poorly executed training event. 
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VI. RECOMMENDATIONS AND CONCLUSIONS 


A. CONCLUSIONS 

Based on the results of Experiment 4, we conclude that serious games 
such as VBS 2™ are valid training tools for small unit tactics. Serious games do 
not necessarily produce better results than traditional training methods, but they 
perform at least as well. Serious games show an edge over traditional training 
methods in user satisfaction, possibly because they satisfy the technical whims 
of today’s youth. From our observations, serious games are not a single answer 
to training needs, but rather a training multiplier to use with other training 
methods such as the sand table or tactical decision game. Serious games are a 
viable tool for the trainer’s toolbox and are well worth the time, money, and effort 
to develop as core assets to a unit’s or school’s tactical training curriculum. 

Serious games come with a price, however. The trainer must understand 
how to use the serious games as an extension of his or her training skills. 
Experiment 5 provided insight into a representative situation that demonstrated 
that a trainer’s use of serious games is a learned and practiced skill rather than a 
natural and intuitive technique. The trainer must be able to clearly articulate 
training objectives. Then, the trainer must understand the capabilities and 
limitations of the serious game well enough to translate the training objectives 
into an implementable serious game exercise. For the serious game training to 
be successful, some combination of people must fill the roles of facilitator, 
technician, and planner/evaluator. The trainer must fill the last role and is better 
served knowing as much as possible about the first two. We did not find 
evidence to support the notion that the trainer’s involvement in the facilitator role 
can improve the training experience, but we think this idea deserves further 
investigation. Flowever, we did find evidence to support the conclusion that the 
trainer cannot focus on the facilitator role to the exclusion of the other roles; such 
a practice leads to a chaotic evolution. 
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We conclude, then, that the technical aspects of serious games are no 
longer the primary issue in providing innovative and effective small unit tactical 
training for Marines and soldiers. The driving issue at hand is developing the skill 
and experience of the trainer to use serious games effectively. We conclude that 
the training establishment must think out such a program carefully, plan it, 
support it, monitor it, and supervise it. A program that simply makes the training 
tool available to the trainer without actively changing the way a trainer develops 
his or her training program risks wasting the potential of the tool, wasting the time 
of the training audience, and wasting the valuable support dollars that brought 
serious games into the limelight. 

A training transfer study seeks to determine whether a given training 
device makes an individual more effective or proficient in a certain task or 
mission. Tradition implies that this analysis should focus on the trainee. 
Throughout the course of this thesis adventure, we did just that with VBS 2™- 
that is, we demonstrated an improvement in self-assessed proficiency within a 
given skill set. More broadly, we determined that serious games are as effective, 
but not necessarily more so, than other training methods. If serious games are 
as good but no better than anything else, then the question boils down to what is 
better for the trainer. We conclude that it is time for the era of the trainee 
focused training transfer study to close for serious games. The focus of training 
transfer study in the serious game domain needs to turn to the trainer. Such a 
study should determine whether the serious game makes the trainer more 
effective in his or her mission. If the trainer cannot achieve training objectives 
more efficiently, serious games do not offer enough game to make them worth 
the time, money and hype. We firmly believe that serious games offer the trainer 
a whole new world of capability, but the trainer must be well educated in the 
employment of the tool to realize this potential. 
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B. OBSERVATIONS 


This thesis provided a unique opportunity for comment on serious games. 
In this project a Marine Corps field grade officer learned to administer training 
with VBS 2™ through experience and self-instruction. All of our experiments 
were done with what we could learn from VBS 2™ as it came out of the DVTE 
boxes. Resources only included the manuals on the computers, the VBS 2™ 
website, and the on-line VBS 2™ support forum. We got a three-day summary 
view of the I MEF Simulation Center contractor-led DVTE training for comparison 
purposes after we had completed the bulk of this project. No outside training 
influenced our work. For this reason, the comments of an experienced trainer 
with the educational background provided by the NFS MOVES program who has 
implemented training using DVTE “out of the box” should have value to the 
tactical training community as well as simulation professionals. 

1. Making VSS2™ an Ubiquitous Trainer 

At one time, in the early 1990s, Marine Corps officers approached 
commercial presentation software, such as Microsoft PowerPoint, with caution 
and trepidation. Officers did not know how to use the software. They were not 
used to seeing information presented with it. Thus, they looked at the software 
with reluctance and passed any duties associated with it to the lowest capable 
individual they could find. Now, most officers use such software quite easily, and 
modern audiences expect presentations developed with such software as a 
means of information exchange and consider these products the norm. This 
change occurred for two main reasons. First, Marines have learned how to use 
the software, at least at a basic level, through experience and observation. 
Second, Marines generally do not have to build presentations from scratch. 
Often, presenters provide information using presentation software formats that 
have been developed and refined over the years, and Marines simply have to 
cut, paste, and update to achieve their objectives. In this way, neither the 
learning curve nor the development curve is steep, so the software is much more 
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palatable. The result is that in today's world, many would not know how to 
accomplish their jobs if the software suddenly disappeared. 

VBS 2™ is at the same point that presentation software was two decades 
ago. Both the learning curve and the development curve are steep. Moreover, 
VBS 2™ is not the method with which people are used to having their training 
presented. The idea that personal computer-based training will one day reach 
the level of ubiquitousness enjoyed by presentation software is not far-fetched, 
however. The question, then, is how to shorten the learning and development 
curves. 

A good way to shorten the learning curve is through schools. The learning 
curve stays long if no one understands what the product is and if nothing is done 
to teach it. The DVTE program provides a training package for unit 
representatives upon dissemination of the gear, but this program does not lend 
itself to ubiquitousness. It only temporarily provides someone who can turn on 
the machines. Without a program in place to continue training, this temporary fix 
will not live. Ubiquitousness occurs through an organized approach that 
systematically touches everyone in the Marine Corps at one point or another. 
Resident schools provide such an opportunity. Resident schools have a 
standardized hierarchy organized through TECOM. Students in resident schools 
have the time and motivation to focus, so instruction does not get lost in a series 
of competing unit demands. Finally, schools can provide students with the 
chance to experiment and ask questions. Not all Marines go to the appropriate 
resident school for their grade. However, if VBS 2™ and other simulations were 
taught in the schools for NCOs, SNCOs, company grade officers, and field grade 
officers, units would have a sprinkling of simulation interest to get personal 
computer-based training out of its infancy. However, the Marine Corps and other 
DoD services must approach this training in schools with discretion. Junior 
leaders do not need to focus on the key that makes the avatar move forward. 
Rather, they need to focus on how to use the simulation as an extension of their 
training capabilities. This includes the technical ability to develop, run, and 
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debrief scenarios as well as the ability to integrate the simulation into the unit’s 
training schedule to provide a complementary medium to enhance the 
achievement of unit objectives. 

The armor community offers an example of this concept in practice. For 
both the Marine Corps and Army M1A1 tank communities, a Master Gunner 
serves as the expert for all matters related to tank gunnery. This enlisted tanker 
goes to an intense school at Fort Knox, Kentucky to learn about all aspects of 
making the tank put main gun and machinegun rounds on target. Part of this 
training includes instruction about the use of the Advanced Gunnery Training 
System (AGTS), the simulator to train the gunner and tank commander in crew 
level gunnery skills. Not only does the Master Gunner learn to use AGTS, but he 
learns to develop a company level gunnery training plan that includes AGTS as a 
core component of the annual training cycle. There is no reason that training 
with serious games cannot follow such a model, and we recommend scrutinizing 
this concept for potential ideas about incorporating VBS 2™ into training 
curricula. 

The training establishment can shorten the development curve through an 
organized proliferation of user generated simulation products. Users should not 
be creating their training from scratch. If one unit puts together a solid simulation 
training package for a certain training objective, there is no reason someone else 
should do the same work over again. However, the creation of an environment in 
which users can pool their work spawns a variety of obstacles that must be 
overcome. First, one must be able to identify what the scenario trains. The 
mechanism for achieving this is not complex or new. The system of MPSs in the 
Marine Corps provides an adequate mechanism for tagging the training 
objectives of a given scenario. Second, one must be able to identify what is in 
the scenario. This information is not difficult to organize since it includes 
standard data that everyone understands such as number of players, number 
and type of vehicles and weapons, type of terrain, and type of enemy situation. 
Third, one must be able to provide quality control for the scenarios. A novice's 
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best work may be a useless product for the experienced trainer. Some authority 
must separate the wheat from the chaff without creating an approval bureaucracy 
so thick that everyone is scared away. Finally, one must provide quick and easy 
accessibility. The information age has solved any technical problem in this 
regard, but the internal administrative issues of military information technology 
are not designed with quick and easy accessibility in mind. Through DVTE, 
Marine Corps VBS 2™ lives in a computer environment entirely cordoned off 
from the Internet environment, and the Navy Marine Corps Intranet (NMCI) does 
not offer easy solutions to this challenge. 

With formalization of standard training packages, the Marine Corps 
training community can implement learning management software to help 
trainers monitor the unit’s progress. We believe that rigid emphasis on such 
tools can stifle training initiative and create programs that merely put a check in 
the box. Rather, a learning management tool should provide trainers with a 
picture of current and desired training levels. Then the tool should recommend 
ways to make up the difference. Serious games such as VBS 2™ lend 
themselves to such management programs, and these programs would help 
institutionalize serious games as credible training platforms. 

The Joint Training Counter-Improvised Explosive Device Operations 
Integration Center (JTCOIC) uses VBS 2™ extensively. The organization has 
collected, organized, and documented its VBS 2™ models, maps, scripts, and 
scenarios and makes them available through regular e-mail dissemination. 
JTCOIC has no charter to act as a DoD VBS 2™ clearinghouse, but their efforts 
at organizing VBS 2™ material provide a model for the rest of the military VBS 
2™ community. Expanding on this model DoD wide could benefit all users 
exponentially. 

2. Making a Serious Game Trainer 

In our Experiment 5 comments, we pointed out that a trainer using VBS 
2™ must be a facilitator, technician, and planner/evaluator or have such 
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personnel on his or her staff. Commanders with DVTE in their inventories must 
consider personnel when determining how to implement the simulations at their 
disposal. Certainly, junior enlisted personnel should be trained to fill the 
technician role of serious game training. Equally as clear, senior SNCOs and 
junior officers should fill the planner/evaluator role. However, commanders 
cannot depend on junior enlisted personnel to fill the facilitator role and must 
ensure that SNCOs and junior officers are suited to either fill this role or closely 
supervise it. We believe the power of virtual training lies in the judicial 
application of the facilitator role and the transfer of this role to the trainer. 
Commanders can guarantee this by getting trainers to DVTE training as well as 
junior enlisted personnel who will serve as “pucksters.” 

3. Appreciation of VBS 2™ in the Fieet 

The generation of officers and upper level federal employees who are 
buying today's training systems are technically adept and progressive enough to 
understand that today's youth expect a technically based training experience 
from the technology revolution. However, they do not necessarily understand 
this wired generation. The exuberance to appease the young serviceman who 
has never lived without Internet or computers has led to a variety of assumptions 
that may not be true and may have never been true. Because of the pace of 
technological change, the truth behind these assumptions may change too fast to 
ascertain their validity at any point in time. 

For example, one might surmise that today's generation would appreciate 
personal computer-based gaming for training. This assumption seems to fit with 
the stereotype of the data-device-wielding teenager. Furthermore, one can find 
numerous data to support the idea that today's youth play video games. 
However, data device use does not connote a gamer, and a gamer does not 
connote appreciation of serious games as training devices. A similar domain 
illustrates this concept. Almost everyone has seen a movie, most people have 
seen many movies, and many people enjoy movies as a form of entertainment. 
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However, that does not mean that Marines enjoy watching military films for 
training. Even if Marines like movies with military themes, they most likely enjoy 
qualities of the movie that lend to entertainment rather than realism. We must 
remember this concept when we consider serious games and today's 
technophilic youth. 

Even for the avid gamer, we cannot assume an immediate appreciation of 
serious games for training. Often, the gamer is the one who dislikes the training 
simulation the most. Most people who do not bicycle would consider any bicycle, 
regardless of style, source, quality, age, or other factors, the same. On the other 
hand, an avid cyclist would certainly never use a mountain bike for road racing 
and would quite likely seek the highest end racing bike money could buy if road 
racing was the objective. Gaming is no different. Different games have different 
purposes, qualities, and styles. Gamers often bring a set of expectations to the 
training simulation that the military cannot satisfy in such a dynamic industry. 
The barracks XBox jockey might be the very person most put off by the personal 
computer-based simulator chosen by the military service. 

In our enthusiastic theorizing about serious games, we might be tempted 
to imagine a day when military personnel train in virtual environments of their 
own volition and initiative because they enjoy the game. Our experience in this 
study showed that, left alone. Marines like to do in a simulation environment what 
they like to do anywhere else when left alone to their own devices. They like to 
fool around. They like to work as independent players, getting the best of their 
mates. They like to have fun. While some of this play may lead to tactical 
lessons learned, unit training can only be realized through a disciplined 
application of training developed according to an organized set of objectives. 
Simulation training cannot eliminate the need for the trainer any more than it can 
eliminate the need for live training. 

We do not record these observations to suggest that personal computer- 

based training is a lost cause. Quite the contrary, today's military is not surprised 

that it exists and readily embraces it. Arguably, they expect it and have high 

184 



standards for it. However, we cannot expect them to appreciate the training 
simply because we were progressive enough to militarize a computer game. 
Today's youth expects both technical quality and training quality from the 
simulation, as they should. This demands an adept trainer who knows how to 
use the simulation as an extension of his training abilities. It also demands a 
simulation architecture that can keep up with the times, modernizing training 
tools at the rate that Marines update the equipment in their barracks rooms and 
living rooms from the local gaming superstore. 

4. Counting dollars 

In Experiment 4, we demonstrated that VBS 2™ can train Marines as 
effectively as traditional means such as the sand table for small unit tactical 
skills. This work shows why training with serious games should be considered in 
a resource constrained environment that precludes live training. However, the 
findings suggest that the trainer can consider traditional training on an equal 
footing with serious game training. Certainly, when it comes down to dollars and 
cents, the operational trainer must choose to buy a suite of computers with a 
vision toward distinct and unique training gain. We believe serious game training 
offers advantages over traditional training garrison exercises. While we do not 
propose to use this thesis as a cost analysis vehicle for serious games, we 
suggest the following advantages of serious game technology. 

a. Users Like It 

Experiment 4 demonstrated that Marines preferred using VBS 2™. 
Subjectively, Marines felt their tactical skills improved more using VBS 2™ than 
using traditional training methods. Despite unique gamer biases, training aligned 
with the modern technical age appeals to young warfighters. Marines and 
soldiers are used to computers in their daily life, and they accept training 
administered through them. As Experiment 5 demonstrated, this acceptance 
cannot be taken for granted; Marines still expect quality training in virtual 
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environments like anywhere else. Marines like to train live, but when this is not 
possible, they like using well administered serious games 

b. The After Action Review Tooi Expands the Trainer’s 
Options 

In live training, the exercise proceeds, and then it is gone. Only 
memories and notes remain after its completion. Traditional training like the 
sand table produces the same results. However, serious games such as VBS 
2™ offer the ability to record the exercise and view it exactly as it happened. As 
a matter of fact, the trainer can review the exercise from many different 
perspectives, focusing on points of interest in whatever manner he chooses. 
This review capability is unique to virtual environments, and it offers the trainer 
capabilities that cannot be achieved through any other means. 

c. Events are More Reai 

Virtual training offers the potential for partial, or possibly complete, 
immersion in the training environment. Trainers cannot achieve anything but the 
most remote level of immersion using sand tables or tactical decision games. 
While virtual training environments cannot achieve enough immersion to rival a 
live exercise, they provide the only opportunity for immersion out of the choices 
for garrison training. Trainees can experience weapons effects, death, the visual 
effects of motion, the sounds of combat, and other battlefield effects. While 
these virtual experiences are certainly far from real, they create a more realistic 
evolution than traditional means. Arguably, some effects, such as death and 
injury, are even more real than live training. This unique version of realism 
demands that serious games be included in tactical training curricula. 

d. Training can be Standardized 

Computers offer dependability that training based on human 
interaction can never achieve. Once a trainer develops a valid training regimen 


186 



in a serious game, the trainer can administer the scenario the same way every 
time. The trainer can export the scenario to other trainers easily. This unique 
advantage of computer-based training provides the opportunity to align training 
scenarios with doctrine and then mass produce them for the convenience of 
many training audiences. Neither live nor traditional training can achieve this 
level of standardization, and trainers can serve their communities well by 
capitalizing on this feature. 

e. Serious Games Offer Ties to Higher Headquarters 

Higher level command cells, particularly battalion and above, tend 
to rely more and more on simulation to achieve their training goals. Mobilizing 
large units for training is costly and manpower intensive. Simulation offers the 
opportunity for large staffs to exercise procedures and techniques while avoiding 
these costs. Serious games allow the small unit tacticians to interface with these 
higher level simulations. Such interoperability sets the stage for a symbiotic 
relationship. Small units have the benefit of operating in a domain supervised by 
a higher headquarters with all of its help and hindrances while higher level units 
have the added chaos of subordinate units making decisions at their level with a 
thinking mind instead of a computer’s artificial intelligence. When the 
Lieutenants and the Lieutenant Colonel need to train together, simulation offers 
an outstanding medium to accomplish goals without wasting the valuable time of 
a whole unit full of Marines or soldiers. 

5. Shopping for Serious Games 

An interest in the potential of transfer studies as decision drivers in the 
acquisition of training games partially motivated this thesis effort. When 
choosing a serious game for training purposes, the military trainer can use four 
methods to select the appropriate platform. First, the trainer can conduct a 
training transfer study like the project documented in this thesis to explicitly 
determine the effectiveness of the platform. Second, one can shop by features. 
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depending on the written catalog descriptions or the verbal explanations of 
vendors to provide the necessary information to make the decision. Third, the 
trainer can conduct a user study in which a representative user sample compares 
several similar platforms and provides comparative subjective evaluations. 
Fourth, one can use the gut instinct of what looks good based on whatever 
experience the trainer has accumulated in the domain. 

Based on this thesis project as a whole, we conclude that the training 
transfer study does not adequately support the serious game acquisition process. 
Quite simply, such a study involves much effort with little return. The process of 
gaining access to user units, developing satisfactory evaluation mechanisms that 
produce scientific results without handcuffing the unit’s initiative, and following 
the project through the chaotic demands of operational military life requires much 
effort, readjustment from missed opportunities, and sheer luck. Furthermore, the 
results tend to be murky at best. For transfer studies in the group training 
domain, even the most rigid transfer studies leave gaping holes for the skeptic to 
launch unanswerable concerns. 

The reader should not interpret the preceding paragraph as a suggestion 
to discard training transfer studies altogether. Training transfer studies 
potentially offer unique and valuable insight to the research community that no 
other methodologies can attain. If nothing else, the process of the research 
project itself results in a well-documented investigation into the specific areas of 
success and failure of the training tool. Flowever, the slow, arduous, and 
potentially inconclusive world of training transfer studies will most likely only offer 
frustration to the decision maker trying to spend Department of Defense dollars 
on a narrowly defined budget, time schedule, and performance objective set. 

The military trainer must decide what systems to buy somehow, and the 

preceding argument only serves to nay-say one method. We did not structure 

this thesis project in any way to answer the question at hand, but we feel the 

lengthy hands-on effort has provided ample qualification to include some notes of 

opinion on the topic. We start the discussion by stating up front that the answer 
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is not some clean-cut checklist style formula. Rather, a sound serious game 
acquisition effort will most likely include some aspect of each of the last three 
selection methods. 

Arguably, feature shopping drives the current serious game acquisition 
process. One simply jots down requirements, reviews vendor descriptions of 
applicable products, and decides according to some designated cost analysis 
criteria. The Interservice/Industry, Training, Simulation, and Education 
Conference (l/ITSEC) offers one of the best opportunities for the military trainer 
to feature shop. Vendors shamelessly showcase their most innovative 
contributions to the simulation community with flashy screens, fancy interfaces, 
and brilliant sound effects. We are not saying l/ITSEC is bad; quite the contrary, 
l/ITSEC plays an absolutely vital role in educating the military training community 
about the available market. Nevertheless, the military professional who treats 
l/ITSEC as a buffet is doomed to come home with an empty pocket and a 
warehouse of gadgets that will gather dust. Common sense dictates that only a 
foolish buyer relies exclusively on the advice of sales professionals. One must 
shop intelligently. Feature shopping is certainly an important part of the process, 
but this sort of selection method executed exclusively will likely result in 
overpriced bells and whistles and a deficit of substance. 

We illustrate the idea of user studies by visiting the Marine Corps’ recent 
efforts to update the individual fighting gear for Marines. Over the past decade 
and a half, the Marine Corps has updated virtually all of the personal gear 
inventory, including packs, sleeping bags, foul weather gear, the utility uniform, 
and boots. For the most part. Marines contentedly use this gear, happy to have 
emerged from the outdated world of shelter halves and rubber rain suits. The 
acquisition program employed a notable technique of user testing for all of this 
gear. Representative Marine units tested the gear in all sorts of conditions, and 
the opinions of warfighting Marines literally determined the course of the 
acquisition effort. This technique produced high quality upgrades that have 
resulted in improvements to warfighting capability and individual morale. 

189 



Certainly, this technique suits the acquisition of serious games as well with one 
striking exception. A Marine testing a sleeping bag knows that he needs a 
product that is light, dry, and warm. He knows this from his experience with other 
sleeping bags. If the Marine had never used a sleeping bag before, he might 
pick a light sleeping bag that left him wet in a rainstorm or the warmest sleeping 
bag that weighed too much for practical transport. The Marine’s experience with 
similar products enables him to successfully contribute to the selection process. 
In the case of serious games, most Marines do not have the requisite experience 
to know what they want or to know when the product is “good.” During this 
project, we noticed that Marines often judged VBS 2™ training either because 
they liked the novel approach or because the non-traditional approach disturbed 
them. Either way, serious games for training fell outside their experience base, 
leaving them unqualified to comment on the quality of the specific platform. 

Military decision making does not always occur with a highly structured 
analytical approach due to time constraints, lack of resources, or other obstacles. 
In the absence of structured analysis, military professionals often make quick 
decisions based on available information and relevant experience. The 
expression “a good seventy percent solution now is better than a perfect one 
hundred percent solution later” sums up this concept. Because military leaders 
employ this technique in training and war, one would not be surprised to see 
them employ it in acquisition. We do not dismiss this technique as inappropriate 
to the acquisition of serious games. Certainly, the instantaneous impression of a 
General or Master Gunnery Sergeant with decades of experience can lend more 
to the selection process in less than sixty seconds than months of analysis. 
However, such a process employed exclusively has neither the analytical power 
required by those holding the purse strings nor the breadth to represent every 
user situation. At its worst, this technique turns into a single individual buying pet 
toys, and at its best, the method results in rapid turn around that catches the user 
base off guard and leaves critical people excluded from the process. 
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For serious game acquisition, this discussion leaves the warfighter in a 
quandary. If the complexities of measuring team cognition preclude a bona fide 
effectiveness study, the user must buy systems intelligently some other way. In 
Section 1, we discussed techniques of making VBS 2™ ubiquitous through the 
Marine Corps’ system of schools. The solution to the serious game acquisition 
effort lies here as well. This thesis demonstrates that serious games can 
effectively contribute to training. Schools provide the means to quickly get 
Marines experienced in the employment of serious games for training. Once 
exposed to serious games and taught how to use them. Marines will develop the 
requisite experience base to support the user studies and gut instinct buying 
approaches that can best select the most suitable training platform out of all the 
competing products on the market. 

6. Conservative Use of Options 

VBS 2™ allows the user to manipulate a variety of options to enhance the 
user experience. In this way, one can control whether a participant dies, how the 
participants' interface works, and a variety of other practical issues. We felt that 
we should alter these options from the defaults with caution. The Marine Corps 
intends to make VBS 2™ a ubiquitous training asset, and Marines will expect the 
training device to look the same and operate the same from exercise to exercise 
and from command to command. To some extent, personalizing training assets 
allows a Marine to get the most out of the gear, but in an environment of constant 
turbulence and rotation, there is value in all assets being the same. We did not 
rule out the idea of tailoring the VBS 2™ options to best support our goals, but 
we took a conservative approach to changing them. 

After conducting several training exercises in the series of pilot studies 
and experiments, we noted some options that we felt necessary to change from 
the default. In interface training, it is helpful to force all avatars to stay alive 
regardless of battlefield damage. Despite all attempts at discipline and caution, 
novice users will accidentally shoot each other. The cost in terms of time and 
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frustration of bringing users back into the environment is not worth the realism for 
initial training. Of course, once initial training is done, this argument does not 
hold. For most training exercises, invincibility would almost certainly teach 
Marines the wrong thing. Invincibility is only useful as a matter of convenience 
when teaching button pushes. 

For the training we conducted, it was helpful to disable the third person 
view. Likewise, the "M" key allows users to pull up the two-dimensional map 
view. While the map is quite useful, it depicts enemy locations, so it is an 
unrealistic asset. We disabled the "M" key function so that users could not see 
this view. In general, we felt that the best training evolved from users only seeing 
the world from the first-person view of their position. With this in mind, we paid 
attention to disable all other views and to position participants so that they could 
only use their own computer screens to see the environment. 

VBS 2™ has a re-spawn capability for avatars that die during the course 
of the scenario. Avatars re-spawn into swallows that can fly around the local 
area of their unit observing the action from a third person view. We enabled this 
function. We found that users who died in the scenario and were doomed to 
stare at a black screen for the remainder of the exercise got little training and, in 
some instances, detracted from the training of others. The re-spawn function is 
quite helpful to remedy this training problem. 

7. Not All Terrains are Created Equal 

During the course of our experiments, we noticed a difference between 
the quality of the Twentynine Palms terrain and the Sahrani terrain. The two 
terrain databases come from different origins. The Twentynine Palms database 
hails from Marine Corps specifications, while the Sahrani database satisfies the 
needs and desires of the gaming community. However, the Twentynine Palms 
terrain did not offer the same quality of training experience as the Sahrani terrain. 
Vehicles in the Twentynine Palms terrain maneuvered on roads as though they 
drove on paved highways instead of the off road trails that exist in reality. 
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Vehicles in the Twentynine Palms terrain could easily climb mountains that they 
could not drive on in reality. The microterrain features that characterize the 
Twentynine Palms landscape, limiting wheeled mobility, were not represented at 
all. Twentynine Palms is full of rugged rocks, wadis, holes, craters, and other 
features that make wheeled movement difficult, or sometimes impossible, and 
wheeled vehicles could move easily across open terrain in the virtual 
environment database. The Sahrani terrain, on the other hand, provided more 
realistic roads, hills, microterrain features, and other features. One can debate 
the level of fidelity needed for useful training, but fidelity must certainly be high 
when it relates to the specific task being trained. Convoy training revolves 
around the mobility of forces, and terrain characteristics dramatically impact this 
mobility. The unit seeking to use Twentynine Palms terrain to support a 
rehearsal scenario could be sorely disappointed. The unit could very well find 
that vehicles could not execute a well planned operation because of the realities 
of the ground. 

In this thesis, we do not specifically seek to define which terrain databases 
are suitable and which are not. We feel it is important that those involved in the 
development and procurement of terrain databases for VBS 2™ evaluate the 
objectives of the terrain. These people must ensure they write the specifications 
for VBS 2™ terrain to support the potential associated training objectives. Desert 
terrain can be as simple as a sand colored plane, and a building can be as 
simple as a textured cube. However, some cubes scattered on a tan plane do 
not necessarily make adequate terrain for the trainee. On the other hand, the 
trainee does not necessarily need to see each blade of grass blowing in the wind 
or the interior of every building in the terrain. The answer lies somewhere 
between the two, and the arbitrary middle point is a distinct function of potential 
training objectives. VBS 2™ terrain database construction offers a very real 
potential for waste of money, either in the purchase of terrain that offers 
unnecessary realism or the development of terrain that is so poor in quality that it 
is never used. 
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8. Communications in 1^85 2™ 

During this project, we experimented with two software applications to 
support communications. We used JVTR and CNR SIM. We found that while 
we could accomplish communications objectives with either, we considered both 
unsatisfactory for the needs of our project. Quite simply, communications should 
be as seamless as all other aspects of VBS 2™. Communications should be 
internal to VBS 2™ with seams between vendor production transparent to the 
user. Our current version of VBS 2™ has five icons on the startup screen for 
functions such as setup, networking, editing, and starting a scenario. A user 
should have an additional icon for communications. From this icon, the user and 
administrator should be able to access appropriate buttons and toggles and then 
start the simulation. The amount of headaches associated with JVTR and CNR 
SIM are not consistent with the rest VBS 2™. Our work in this project 
demonstrated consistently through all five experiments that communications is 
fundamental to the training experience and one of the most important aspects of 
the exercise. The communications support for VBS 2™ needs immediate 
attention. 

9. Computer Glitches in VBS 2™ 

We noticed a few computer glitches in VBS 2™ that bear mention. First, 
our work in the interface familiarization pilot study discussed in Chapter III 
revealed that lED disarmament disrupts the AAR recording. As we have 
mentioned previously, the AAR often provides the bulk of the training value for a 
VBS 2™ exercise. If lED disarmament plays a critical role in the scenario, this 
problem could limit the training value of the simulation exercise. Likewise, we 
discussed the problem of a driver and gunner losing control of their avatars in 
some circumstances in networked scenarios. We found that this problem could 
be remedied by switching into map view and back out again, but this problem 
inhibits training. Worse, it usually occurs right in the heat of some action that is 
the center of the training objective. 
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We understand that computer glitches such as these have been identified 
to Bohemia Interactive and will be fixed in later editions of VBS 2™. VBS 2™ 
offers a Web site forum for the discussion of such topics. In this forum, users 
from every sector of the VBS 2™ community post lessons learned and interact 
with the VBS 2™ support staff to come up with the best possible answers. This 
support forum is vital to the ultimate success of VBS 2™, because it allows the 
software package to be a living thing, continuously adapting to the needs of the 
user. However, it also requires the user to stay attuned to the latest information 
to provide the best possible training for the unit. Many VBS 2™ questions have 
been asked, and many answers are known, but ensuring the Marine with the 
question reaches the applicable answer can be a daunting task. 

10. Training Notes 

We collected a few miscellaneous notes that we feel worthy to document 
for anyone seeking to follow in our footsteps working with VBS 2™ as a training 
platform. The trainer must consider time. The simplicity of a laptop trainer may 
tempt one to believe that training can occur on a dime. This is not true, at least 
initially. We found that twelve computers require approximately one man hour to 
set up and one man hour start up. This rule of thumb can be circumvented by 
training the Marines using the system to do the set up and start up procedures, 
but this requires its own amount of time and potentially subjects the equipment to 
abuse that is difficult to see and control. Additionally, even the most dedicated 
Marine can only train with VBS 2™ so long before it turns into a game, and a 
monotonous one at that. Limited training times seem to promote the best 
training. 

The trainer must seriously consider how to ensure trainees take the 
simulation seriously. It does not take much for VBS 2™ to transform from 
training into a game. However, this is true of live training, or any other training, 
as well, and the solution to the problem, whether live or simulated, is leadership. 
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Without the proper leadership in place, the trainer will find that negligent 
discharges, fratricides, and other practices associated with games rather than 
training will dominate the exercise. 

VBS 2™ is not only a suitable forum for observing trainees, but it provides 
an opportunity to observe the trainer. Often, the trainer is as much of a novice at 
training Marines as the Marines are at conducting the training. For the Company 
Commander seeking insight into how his Platoon Commander trains his Marines 
or for the Platoon Commander trying to show his new Platoon Sergeant how to 
train, VBS 2™ provides an excellent opportunity to discuss what is important and 
how to present it to the training audience. 

C. RECOMMENDATIONS FOR FUTURE WORK 

1. Repeat Experiment 5 to See the Effects of the Trainer 

In Experiment 5, we unsuccessfully attempted to compare simulation 
training exercises in which the trainer implemented training through a third party 
with exercises administered by the trainer alone. However, our observations 
provided ample evidence that this domain of study bears further investigation. If 
a researcher could convince a unit to train Marines in different venues, a 
comparison study using the unit’s DVTE assets versus the same assets in a 
Simulation Center would shed light on this matter. We believe this question is 
highly important, because it drives decisions about whether the Marine Corps 
and other services should field and maintain serious games through operational 
units or only house and administer them through contractor staffs in base 
Simulation Centers. 

2. Can \^SS2™ Scenarios be Catalogued and Classified? 

We noted that VBS 2™ could move toward ubiquitousness as the user 
base develops a body of tried and tested scenarios to pass throughout the user 
community. The less Marines have to invent the wheel, the more inclined they 
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will be to use the tool. Right now, there is no good mechanism to pass VBS 2™ 
scenarios around in a structured, organized way. In order for one Marine to use 
another Marine’s scenario, the Marine must know the purpose of the scenario, 
the training objectives it should fill, the number of players involved, the type of 
support involved, and a variety of other information. JTCOIC has made great 
strides in documenting how they made their scenarios, what they do, and what 
they look like and has even started including video to support the documentation. 
Future work could investigate Best Practices for formalizing such documentation 
so that the operator could select VBS 2™ scenario support from a menu, easily 
matching scenario characteristics to training needs. 

3. Do Peripheral Computer Devices Enhance VBS 2™ Training? 

Currently, the Marine Corps fields VBS 2™ with the idea that it will only be 
used on a laptop with the standard keyboard and mouse interface. One can alter 
the interface by adding game controllers, larger screens, head mounted displays, 
more realistic communications interfaces, and other peripheral devices. VBS 2™ 
derives a certain amount of benefit from its easy deployability; at the end of the 
day, a Marine only needs to move around a laptop and some cables. Future 
work could investigate the addition of some of these peripheral devices to 
determine whether they add enough training benefit to justify their cost and 
added logistical burden. 

4. Would a VBS 2™ Filming Capability be Useful for Training? 

VBS 2™ shows potential for use as a demonstration tool as well as a 
practical application tool. A trainer can choreograph an exercise with relative 
ease to show Marines what “right” looks like. This idea is not novel; the Army 
used DARWARS Ambush! in this mode (Roberts, Diller, & Schmitt, 2006), and 
JTCOIC has produced videos to demonstrate its products. Flowever, one cannot 
easily film VBS 2™ exercises for demonstration purposes. The trainer must 
figure out how to use an external screen capture tool and must do all video 
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editing externally. Future work could investigate whether an internal VBS 2™ 
filming function could provide training benefit and how this video function should 
be added to the software. 

5. When Is the Third-person View Preferabie to the Map View for 
the Real-time Editor and After-action Review Functions? 

We noticed that different trainers prefer different views for using the real¬ 
time editor and the AAR tool. While the flexibility is convenient for different 
people to use VBS 2™ in their own way, a better understanding of when each 
view is most useful would be helpful in training the trainer to make the best use of 
the simulation. Not only must the trainer know which view is best to personally 
look at the situation and affect training, but the trainer must know the best view to 
show different battlefield activities when debriefing exercise participants. Future 
work could investigate how a trainer can best use these different views to provide 
the most training value. 

6. Is There a Potential for Civilian Users of Armed Assault to 
Contribute to Marine Corps VSS2™ Training Efforts? 

In Chapter II, we provided an extensive review of gaming throughout 
history highlighting the continuing link between civilian leisure gaming and 
military training. VBS 2™ has a rich history in the civilian gaming industry. VBS 
2™ originated from the Operation Flashpoint game engine, and the current 
civilian version of the game. Armed Assault, is relatively interchangeable with 
VBS 2™. VBS 2™ can use models, terrain, scenarios, and scripts developed for 
Armed Assault. Likewise, Armed Assault can operate with VBS 2™ creations. 
The Armed Assault community is a vibrant group with extensive interaction. Web 
sites, such as www.armaholics.com, allow Armed Assault users to share their 
products and benefit from each others’ experience. Typically, Armed Assault 
users seek realism in their gaming and might have an appreciation for something 
that is very real: military gaming. For example, while the Marine Corps and 
other DoD agencies have worked on building realistic Afghan terrain databases, 
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Afghan terrain has shown up on www.armaholics.com. Taken to the extreme, 
civilian Armed Assault players could present the ultimate oppositional force by 
playing Marines or soldiers in their own game. Future work could explore 
whether the military could leverage the civilian gaming community for mutually 
beneficial products, thereby gaining training assets for free. 

7. Who Are this New Computer Generation and What Do They 
Expect? 

The simulation training community has been quick to tout simulation training 
as a way to capitalize on the technical expertise of the youth filling our junior 
enlisted ranks, who have never lived in a time without computers. We have 
discussed the idea that these young people may not all have the gaming expertise 
or appreciation for gaming that we think they do. Moreover, this appreciation of 
technology may not spread equally across ranks, age groups, military services or 
components, or job specialties. For example, a young communicator reservist 
may appreciate serious games for training more than an active duty infantryman 
regularly training in the field. We have seen enough of this issue discussed first¬ 
hand to highly recommend a demographic mapping of the Marine Corps in terms 
of computer gaming expertise and appreciation. An organized pursuit of serious 
games for training’s sake hinges on an understanding of the people who will use it. 
We cannot properly design interfaces, train users, write manuals, or field systems 
without a better understanding of the Marine sitting at the keyboard. This study 
has the potential to extend far beyond the bounds of serious games, as the DoD 
investigates the potential of unmanned equipment on the battlefield, bringing 
serious gaming and reality in dangerous proximity to each other. 
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APPENDIX A. INFORMED CONSENT (EXPERIMENT 1) 


Informed Consent Form 

Introduction. You are invited to participate in a research study entitled Virtual Battlespace 2 User 
Familiarity Pilot Study. This study supports a project to compare the training effectiveness of a 
first-person shooter simulation with traditional tactical training methods such as the sand table. 
This pilot study tests procedures to familiarize users with the simulation. 

Procedures. The simulation familiarization will consist of the following: 

Survey to better understand the user’s level of computer expertise; 

Overview brief describing the project and the pilot study’s purpose; 

User interface brief describing the basic functions of the simulation; 

User experimentation period, where participants will be able to freely test the 
ideas just presented; 

User evaluation, where participants will complete a short exercise demonstrating 
their skill level with the simulation. 

The pilot study will take no longer than the 50-minute class period. 

Risks. The potential risks of participating in this study are not greater than minimal risk. The 
study involves no known reasonably foreseeable risks or hazards greater than those encountered 
in everyday life. 

Benefits. The anticipated benefit from this study is gaining insight into the viability of first-person 
shooter simulations as tactical training devices. 

Compensation. No tangible compensation will be given. A copy of the research results will be 
available at the conclusion of the experiment. If you would like a copy of the results, e-mail Major 
Ben Brown at bjbrown@nps.edu. 

Confidentiality & Privacy Act. Any information that is obtained during this study will be kept 
confidential to the full extent permitted by law. All efforts, within reason, will be made to keep 
your personal information in your research record confidential but total confidentiality cannot be 
guaranteed. No information which could identify a participant will be publicly accessible. 
Records of participation will be maintained by NPS for 3 years, after which they will be 
destroyed. However, it is possible that the researcher may be required to divulge information 
obtained in the course of this research to the subject’s chain of command or other legal body. 
Voluntary Nature of the Study. Participation in this study is strictly voluntary, and if agreement 
to participation is given, it can be withdrawn at any time without prejudice. 

Points of Contact. It is understood that should any questions or comments arise regarding this 
project, or a research related injury is received, the Principal Investigator, Dr. William J. Becker, 
656-3963, wibecker@nps.edu should be contacted. Any other questions or concerns may be 
addressed to the Navy Postgraduate School. IRB Chair, LCDR Paul O’Connor , 831-656-3864, 
peoconno @ nps.edu. 

Statement of Consent. I have read the information provided above. I have been given the 
opportunity to ask questions and all the questions have been answered to my satisfaction. I have 
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been provided a copy of this form for my records and I agree to participate in this study. I 
understand that by agreeing to participate in this research and signing this form, I do not waive 
any of my legal rights. 


Participant’s Signature 


Date 


Researcher’s Signature 


Date 
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APPENDIX B. IRB REQUEST (EXPERIMENT 1) 



William J. Becker, Ph.D. 
MOVES Institute 
Watkins Hall 

Naval Postgraduate School 
Monterey, California 93943 


wjbecker@nps.edu 


Fax: 831-656-7599 


831-656-3963 
DSN: 756-3963 


To: Protection of Human Subjects Committee 


Subject: Application for Human Subjects Review (Title): Virtual Battlespace 2 User Familiarity 
Pilot Study 

Projected Start Date: 02 / 16 / 2009 


Month Day 


Year 


I am requesting approval of the attached experimental protocol. The following documentation is 

provided in support of my application. 

The Principal Investigator understands and accepts the following 
ohligations to protect the rights and welfare of research subjects in this study: 

• I recognize that as the Principal Investigator it is my responsibility to ensure that this research 
and the actions of all project personnel involved in conducting this study will conform with the 
IRB approved protocol and IRB requirements/policies. 

• I recognize that it is my responsibility to ensure that valid informed consent / assent (unless 
explicitly waived by the IRB) has been obtained from all research subjects or their legally 
authorized representatives. I will ensure that all project personnel involved in the process of 
consent are trained properly and are fully aware of their responsibilities relative to obtaining 
informed consent / assent according to the IRB guidelines. 

• I will ensure all personnel involved in this study have completed the required IRB Training. 

• I will not initiate any change in protocol without IRB approval. 

• I have no conflict of interest negating me from performing this research. 

• I will maintain all required research records on file; and I recognize that the IRB is authorized 
to inspect these records at any time. 

• I will immediately inform the IRB Chair and NPS Dean of Research of any untoward event or 
injury that involves a research participant. 

• I understand that in the absence of a continuing review and approval, this research may not 
continue beyond the end of the approval period. 

• At the completion of this project, an End-of-Experiment Report will be submitted. 

• I will not commence this research, including subject recruitment, until I have received my NPS 
IRB application approval letter. 


(Signature of Principal Investigator) 


Dr. William J. Becker 
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Application for Human Subjects Review NPS IRB Number: 


Principal Investigator(s): 
Co- PI(s) 


Dr. William J. Becker, Professor, 831-656-3963 
Major Benjamin J. Brown, Student, 831-656-3812 


Title of Experiment: Virtual Battlespace2 User Familiarity Pilot Study 


Approval Requested 


[ X ] New [ ] Continuing [ ] Amendment 


Requested Level of Risk [ ] Exempt [ X ] Minimal [ ] More than Minimal 
Justification: The participants will be asked to use virtual training simulation on laptop computers, 
which is a standard practice for this curriculum. _ 


Work to be done in (Site/Bldg/Rm) 
Watkins 275 / 285 

Estimated date of completion (not to exceed one year from 
start date): 30 Sept 2009 

Maximum number of subjects: 

50 

Estimated length of each subjects participation: 

50 minutes 


Special Populations that will be Used as Participants: 


[ ] Subordinates [ ] Minors [ X ] NPS Students [ ] Special Needs (e.g. Pregnant women) 

Specify safeguards to avoid undue influence and protect subject’s rights: 

Participation is voluntary, and each participant will have the option to withdraw from the 
study at any time. 

Participants will be informed that class credit is not involved in the study either for 
participation or for lack of it. 

The data collected in the study will be safeguarded as requested by generally accepted IRB 
standards: each person will be identified only as a code number on all research forms and 
databases; the name of any person on any signed document will not be paired with the code 
number in order to protect identity; and records of the participant’s participation will be 

_ maintained by NPS for 3 years, after which they will be destroyed. _ 

Scientific Merit Review (Check all that apply) 

[ X ] This research is part of a funded project (Job Order Number: R9554 ) 

[ ] This research is a student thesis (Attach a copy of the approved thesis proposal) 

[ ] Other (Attach a complete research proposal - Dept. Chair must sign Application Cover Letter) 

Outside Cooperating Investigators and Agencies: 

None 

[ ] A copy of the cooperating institution’s POC and CO’s approval is attached. _ 

Description of Research: (attach an additional sheet if needed). The purpose of the pilot study is to 
test a procedure to familiarize users with Virtual Battlespace 2 (VBS 2). VBS 2 is a personal 
computer-based first-person shooter simulation used by the Marine Corps. The simulation is 
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installed under contract on Dell XPS computers and deployed in suites of 16 to operational infantry 
units. NPS has 6 such computers, and an additional suite of 16 have been obtained for this project. 
These computers will be used for the pilot study. The pilot study anticipates a training transfer 
study of Virtual Battlespace 2 using a platoon-level convoy exercise. 

Simulation familiarization will consist of the following: 

Survey to better understand the user’s level of computer expertise; 

Overview brief describing the project and the pilot study’s purpose; 

User interface brief describing the basic functions of the simulation; 

User experimentation period, where participants will be able to freely test the ideas just 
presented; 

User evaluation, where participants will complete a short exercise demonstrating their skill 
level with the simulation. 

For the study, students will be randomly assigned to groups of 3 to participate as a HMMWV crew 
in the positions of vehicle commander, driver, and gunner. Tasks to be evaluated include driving, 
navigating, mounting and dismounting the vehicle, employing the vehicle’s machine gun, and 
employing personal weapons. The pilot study will take no longer than a 50-minute class period. 
Method of Subject Recruitment: (attach an additional sheet if needed). Professor John Falby has 
offered to solicit his MV3922 class for volunteers for the pilot study. The class is a seminar 
environment, so students will have no pressure to participate in the study for class credit. 
Additional volunteers will be recruited by e-mail from the remaining pool of MOVES students. 


I have read and understand NPS policy on the Protection of Human Subjects. If there are any 
changes in any of the above information or any changes to the attached materials, I will suspend the 
experiment until I obtain new IRB approval. 

SIGNATURE_ DATE_ 

SIGNATURE_ DATE_ 
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APPENDIX C. INFORMED CONSENT (EXPERIMENTS 2 AND 3) 


Informed Consent Form 

Introduction. You are invited to participate in a research study entitled Virtual Battlespace 2 
Scenario Testing Pilot Study. This study supports a project to compare the training effectiveness of 
a first-person shooter simulation with traditional tactical training methods such as the sand table. 
This pilot study tests computer simulation scenarios used for the tactical training. 

Procedures. The simulation familiarization will consist of the following: 

Survey to better understand the user’s level of computer expertise, relevant 
tactical experience, and attitudes toward simulation for tactical training; 

Overview brief describing the project and the pilot study’s purpose; 

User interface brief describing the basic functions of the simulation; 

User experimentation period, where participants will be able to freely test the 
ideas just presented; 

User evaluation, where participants will complete a short exercise demonstrating 
their skill level with the simulation. 

Overview brief of tactics, techniques, and procedures relevant to the computer 
exercise. 

Simulation exercise to train small unit convoy immediate action drills. 

Simulation exercise to test small unit convoy immediate action drills. 

Surveys to determine the user’s impression of the simulation training. 

The pilot study will take no longer than one training day (8 hours). 

Risks. The potential risks of participating in this study are not greater than minimal risk. The 
study involves no known reasonably foreseeable risks or hazards greater than those encountered 
in everyday life. 

Benefits. The anticipated benefit from this study is gaining insight into the viability of first-person 
shooter simulations as tactical training devices. 

Compensation. No tangible compensation will be given. A copy of the research results will be 
available at the conclusion of the experiment. If you would like a copy of the results, e-mail Major 
Ben Brown at bjbrown@nps.edu. 

Confidentiality & Privacy Act. Any information that is obtained during this study will be kept 
confidential to the full extent permitted by law. Simulation sessions will be recorded using the 
system After Action Review tool, but the recording only shows generic avatar representations of 
participants thereby guaranteeing anonymity. All efforts, within reason, will be made to keep 
your personal information in your research record confidential but total confidentiality cannot be 
guaranteed. No information which could identify a participant will be publicly accessible. 
Records of participation will be maintained by NPS for 3 years, after which they will be 
destroyed. However, it is possible that the researcher may be required to divulge information 
obtained in the course of this research to the subject’s chain of command or other legal body. 
Voluntary Nature of the Study. Participation in this study is strictly voluntary, and if agreement 
to participation is given, it can be withdrawn at any time without prejudice. 
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Points of Contact. It is understood that should any questions or comments arise regarding this 
project, or a research related injury is received, the Principal Investigator, Dr. William J. Becker, 
831-656-3963, wibecker@nps.edu should be contacted. Any other questions or concerns may be 
addressed to the Navy Postgraduate School. IRB Chair, LCDR Paul O’Connor , 831-656-3864, 
peoconno @ nps.edu. 

Statement of Consent. I have read the information provided above. I have been given the 
opportunity to ask questions and all the questions have been answered to my satisfaction. I have 
been provided a copy of this form for my records and I agree to participate in this study. I 
understand that by agreeing to participate in this research and signing this form, I do not waive 
any of my legal rights. 


Participant’s Signature 


Date 


Researcher’s Signature 


Date 
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APPENDIX D. IRB REQUEST (EXPERIMENTS 2 AND 3) 



William J. Becker, Ph.D. 
MOVES Institute 
Watkins Hall 

Naval Postgraduate School 
Monterey, California 93943 


wjbecker@nps.edu 


Fax: 831-656-7599 


831-656-3963 
DSN: 756-3963 


To: Protection of Human Subjects Committee 


Subject: Application for Human Subjects Review (Title): Virtual Battlespace 2 Scenario 
Testing Pilot Study 

Projected Start Date: 04 / 16 / 2009 


Month Day 


Year 


I am requesting approval of the attached experimental protocol. The following documentation is 

provided in support of my application. 

The Principal Investigator understands and accepts the following 
ohligations to protect the rights and welfare of research subjects in this study: 

• I recognize that as the Principal Investigator it is my responsibility to ensure that this research 
and the actions of all project personnel involved in conducting this study will conform with the 
IRB approved protocol and IRB requirements/policies. 

• I recognize that it is my responsibility to ensure that valid informed consent / assent (unless 
explicitly waived by the IRB) has been obtained from all research subjects or their legally 
authorized representatives. I will ensure that all project personnel involved in the process of 
consent are trained properly and are fully aware of their responsibilities relative to obtaining 
informed consent / assent according to the IRB guidelines. 

• I will ensure all personnel involved in this study have completed the required IRB Training. 

• I will not initiate any change in protocol without IRB approval. 

• I have no conflict of interest negating me from performing this research. 

• I will maintain all required research records on file; and I recognize that the IRB is authorized 
to inspect these records at any time. 

• I will immediately inform the IRB Chair and NPS Dean of Research of any untoward event or 
injury that involves a research participant. 

• I understand that in the absence of a continuing review and approval, this research may not 
continue beyond the end of the approval period. 

• At the completion of this project, an End-of-Experiment Report will be submitted. 

• I will not commence this research, including subject recruitment, until I have received my NPS 
IRB application approval letter. 


(Signature of Principal Investigator) 


Dr. William J. Becker 
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Application for Human Subjects Review NPS IRB Number: 


Principal Investigator(s): 
Co- PI(s) 


Dr. William J. Becker, Professor, 831-656-3963 
Major Benjamin J. Brown, Student, 831-656-3812 


Title of Experiment: Virtual Battlespace2 Scenario Testing Pilot Study 


Approval Requested 


[ X ] New [ ] Continuing [ ] Amendment 


Requested Level of Risk [ ] Exempt [ X ] Minimal [ ] More than Minimal 
Justification: The participants will be asked to use virtual training simulation on laptop computers, 
which is a standard practice for this curriculum. _ 


Work to be done in (Site/Bldg/Rm) 
Unit’s classroom on site 

Estimated date of completion (not to exceed one year from 
start date): 31 Dec 2009 

Maximum number of subjects: 

100 

Estimated length of each subjects participation: 

8 hours 

Special Populations that will be Used as Participants: 


[ ] Subordinates [ ] Minors [ ] NPS Students [ ] Special Needs (e.g. Pregnant women) 

Specify safeguards to avoid undue influence and protect subject’s rights: 

Participation is voluntary, and each participant will have the option to withdraw from the 
study at any time. 

Participants will be informed that class credit is not involved in the study either for 
participation or for lack of it. 

The data collected in the study will be safeguarded as requested by generally accepted IRB 
standards: each person will be identified only as a code number on all research forms and 
databases; the name of any person on any signed document will not be paired with the code 
number in order to protect identity; and records of the participant’s participation will be 
maintained by NPS for 3 years, after which they will be destroyed. 

The simulation system After Action Review tool will be used to record simulation sessions. 

_ The tool uses generic avatar representations of participants thereby guaranteeing anonymity. 

Scientific Merit Review (Check all that apply) 

[ X ] This research is part of a funded project (Job Order Number: R9554 ) 

[ ] This research is a student thesis (Attach a copy of the approved thesis proposal) 

[ ] Other (Attach a complete research proposal - Dept. Chair must sign Application Cover Letter) 


Outside Cooperating Investigators and Agencies: 

None 

[ ] A copy of the cooperating institution’s POC and CO’s approval is attached. 
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Description of Research: (attach an additional sheet if needed). The purpose of the pilot study is to 
test small unit tactical convoy training scenarios in Virtual Battlespace 2 (VBS 2). VBS 2 is a 
personal computer-based first-person shooter simulation used by the Marine Corps. The simulation 
is installed under contract on Dell XPS computers and deployed in suites of 32 to operational 
infantry units. NPS has one of these 32 computer suites for this project. These computers will be 
used for the pilot study. The pilot study anticipates a training transfer study of Virtual Battlespace 
2 using a platoon-level convoy exercise. 

Simulation familiarization will consist of the following: 

Survey to better understand the user’s level of computer expertise, relevant tactical 
experience, and attitudes toward simulation for tactical training; 

Overview brief describing the project and the pilot study’s purpose; 

User interface brief describing the basic functions of the simulation; 

User experimentation period, where participants will be able to freely test the ideas just 
presented; 

User evaluation, where participants will complete a short exercise demonstrating their skill 
level with the simulation. 

Overview brief of tactics, techniques, and procedures relevant to the computer exercise. 
Simulation exercise to train small unit convoy immediate action drills. 

Simulation exercise to test small unit convoy immediate action drills. 

Surveys to determine the user’s impression of the simulation training. 

For the study. Marines will be assigned to crews of 3 to participate as a HMMWV crew in the 
positions of vehicle commander, driver, and gunner. Vehicles will operate in convoys of 5 to 9 
vehicles The pilot study evaluates three convoy training scenarios: a user familiarity scenario, an 
immediate action drill training scenario, and a computer simulation replication of a live Marine 
Corps convoy training course. The study uses the simulation’s After Action Review tool and user 
surveys to determine the effectiveness of the training scenarios and troubleshoot scenario design for 

future work. The pilot study will take no longer than one training day. _ 

Method of Subject Recruitment: (attach an additional sheet if needed). A Marine Corps unit will be 
solicited for volunteers for the pilot study. Participant solicitation will be effected through the 
unit’s leadership. Active duty and reserve units will be recruited by personal contact to the 
command staff of each unit. Command staff will decide who, what, and where units will 
participate. _ 


I have read and understand NPS policy on the Protection of Human Subjects. If there are any 
changes in any of the above information or any changes to the attached materials, I will suspend the 
experiment until I obtain new IRB approval. 

SIGNATURE_ DATE_ 

SIGNATURE_ DATE_ 
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APPENDIX E. VBS 2™ CHEAT SHEET (EXPERIMENT 1) 


General I 

Move forward 

w t 

Move backward 

s i 

Move left 

A ^ 

Move right 

D ^ 

Action Menu 

[ 

INDIVIDUAL 

Run 

Shift W 

Free look 

L Alt + L Alt 

Lean left 

Q 

Lean right 

E 

Crouch 

X 

Prone 

z 

Stand 

c 

PERSONAL GEAR 

Fire weapon / Throw grenade 

Left click 

Cycle weapons 

Space bar 

Reload 

R 

Sights 

V 

Lower / Raise weapon 

L Ctrl + L Ctrl 

Time 

T 

GPS 

R Ctrl + M 

Night vision 

N 

Binoculars 

B 

Compass 

G 

VEHICLE 

Interact with vehicle 

U 

Forward, low gear 

Q 

Forward, fast gear 

E 

Brake 

S (Back up) 

Get out of the vehicle 

H + H 

OBJECTS 

Recover casualty 

[ + “Carry body” 

[ + “Put body into 
vehicle” 

Recover vehicle 

Get in front (< 15 m) 

U + Tow icon 

Disable lED 

[ + “Disarm bomb” 
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APPENDIX F. VBS 2™ CHEAT SHEET (EXPERIMENTS 2 - 5) 


MOVEMENT 

Move 

Forward 

W 

Move 

Backward 

S 

Move Left 

A 

Move Right 

D 

Action 

Menu 

[ 

Run 

Shift + W 

Look 

Around 

Alt+ L 

Lean Left 

Q 

Lean Right 

E 

Crouch 

X 

Prone 

z 

Stand 

c 


VEHICLE 

Interact with 
vehicle 

U 

Forward, low 
gear 

Q 

Forward, 

fast 

gear 

E 

Brake/ 

Backup 

S 

Get out of 
the 

vehicle 

H + H 


PERSONAL GEAR 

Fire Weapon 

/ 

Throw 

Grenade 

Left click 

Change 

Weapon 

Space bar 

Reload 

R 

Sights 

V 

Lower / 

Raise 

weapon 

Ctrl + L 

Time 

T 

GPS 

Ctrl + M 

Night Vision 

N 

Binoculars 

B 

Compass 

G 
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OBJECTS 


Recover 

casualty 

(1) [ + select ‘‘Carry body” 

(2) [ + select “Put body into vehicle” 

Recover 

Vehicle 

(1) Get in front of vehicle < 15 m 

(2) U + select tow icon 
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APPENDIX G. FAMILIARITY PILOT STUDY PREPARATION 


Computer setup 

> 6 groups of 2 computers: 

■ Connect each box of 4 computers together on a table with 
the box’s switch 

> On the screen of each computer is a number: 113-X; X is the 
computer number 
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> Use 5’ and 12’ cables to connect to switch 

> Only unwind outlet end of power cords for each computer 

> Do not unwind mouse cord; plug in as is 

> Use one power strip for each set of computers 

> Start all computers with “Instructor”, password InstructDVTE 

■ Odd computers: Start VTK Admin 

■ Even computers: Start VTK User; Fill in nickname as 
computer number 

> All computers to Networking 
Starting the scenario 

> Odd computers: Go to “New” and click on “Familiarity” 

> Even computers: Join the odd computer’s scenario (make sure 
it is the computer in its pair 

> Odd computers: Click OK and when Even comes up Continue 

> Note that all 4 computers will show up together with two 
separate Familiarity scenarios 

Monitoring the scenario 

> When evaluation starts, start recording on odd computer (ESC- 
Real Time Editor-Click red button in upper right corner) 

> If someone gets the map, they clicked the M key; hit ESC to get 
back to normal user view 

Cleanup 

> SAVE TFIE AAR: ESC-Real Time Editor-Click red button in 
upper right corner 

■ Save as Pilot_Familiarity_X_Y where X and Y are the two 
computer numbers 

> Ensure a survey form is collected for each computer 

> Wind up all cords 

> Ensure computers go back to proper box 
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APPENDIX H. FAMILIARITY PILOT STUDY QUESTIONNAIRE 


Please fill in the following questionnaire. All information will be held confidential. If you 
need to expand any answer, please use the comments section on the reverse side of your 
paper. 

1. Were you required to use computer-based simulations as part of your training in the past? 

NO/YES (circle one) 

If YES, what simulations have you used, and what skills were they used to train (for example, 
first-person shooter, flight simulator, operational tactics, etc.)?_ 

2. Do you have experience playing commercial video games? 

NO/YES (circle one) 

If YES, what games have you played, and what kind of games were they (for example, first- 
person shooter, flight simulation, sports, fantasy, etc.)?_ 

If YES, how often do you use video games? 

Approximately_hours per day / week / month / year (circle one) 

3. Do you own a personal computer? 

NO/YES (circle one) 

If YES, how often do you use the computer? 

Approximately_hours per day / week / month / year (circle one) 

4. What hand do you use to operate a computer mouse? 

LEFT / RIGHT / EITHER (circle one) 

5. How often do you use the following computer applications? (please answer all that apply) 

a. E-mail: Approximately_hours per day/week/month/year (circle one) 

b. Browse web: Approximately_hours per day / week / month / year (circle one) 

c. Video: Approximately_hours per day/week/month/year (circle one) 

d. Music: Approximately_hours per day/week/month/year (circle one) 

e. MS Word: Approximately_hours per day/week/month/year (circle one) 

f. MS Other: Approximately_hours per day / week / month / year (circle one) 

6. Participant Number: _ 

7. Date: _ 

8. Year of birth: _ 

9. Service component: _ 

10. Primary MOS or job specialty: _ 

11. Rank: _ 

12. Time in service: _years_months ** TURN PAPER OVER 
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13. Circle your role in the evaluation exercise: DRIVER GUNNER (circle one) 

14. How many targets did you see in the evaluation? Personnel_ Vehicle_ 

15. Without consulting the cheat sheet, what key do you use to drive a vehicle forward?_ 

16. Briefly describe the steps to tow a vehicle. _ 


17. What task was hardest for you? 


18. What task was easiest for you? 


19. Rate your confidence in doing the following tasks in the simulation by checking one block for 
each task (1 means you are NOT confident; 5 means you are HIGHLY confident): 


Body movements (walking, lying in prone, etc.) 

Vehicle movements 
Shooting your rifle 
Shooting vehicle mounted machine gun 
Recovering a casualty 
Recovering a vehicle (towing) 

Disarming an lED 

20. Do you have any suggestions for improving user familiarization training? 


1 

2 

3 

4 

5 





































Comments: 


Thank you for volunteering to participate in this study. 
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APPENDIX I. FAMILIARITY FOLLOW UP EVALUATION 


Participant number 

When instructed to begin, perform the following tasks 
in order: 

1. Shoot the enemy combatant. 

2. Board HMMWV 

3. Drive to enemy combatant. 

4. Dismount HMMWV. 

5. Load enemy combatant’s body into HMMWV. 

6. Engage truck target with grenade. 

7. Assume prone position. 

8. Engage near Echo silhouette target in semi¬ 
automatic rifle mode. 

9. Engage far Echo silhouette target in semi¬ 
automatic rifle mode. 

10. Stand up. 

11. Use simulation watch and write time: 
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APPENDIX J. MOVES PILOT STUDY ROOM SETUP 
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APPENDIX K. WFTBN PILOT STUDY SCHEDULE 


0800-0815 

Administration 

Informed Consent 
Demographic Questionnaire 

0815-0915 

Familiarity 

0915-0925 

Break 

0925 - 0930 

Familiarity Test 

0930 - 0950 

Convoy Express 101 

0950- 1005 

Knowledge Test 1 

1005- 1015 

Break 

1015-1030 

Mission Brief 

1030- 1045 

Mission Planning 

1045- 1100 

Training Wheels 

1100-1130 

Noble Pass 

1130- 1230 

Lunch 

1230- 1245 

Debrief Noble Pass 

Post Questionnaire 1 

1245- 1300 

Mission Planning 

1300- 1400 

Rainbow Canyon 

1400- 1430 

Reward Exercise 

Catch up 

Gays Pass 

1430- 1450 

Knowledge Test 2 

Post Questionnaire 2 
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APPENDIX L. DEMOGRAPHIC SURVEY (EXPERIMENTS 2 - 3) 


Please fill in the following questionnaire. All information will be held 
confidential. 

1. Were you required to use computer-based simulations as part of your training in the past? 

NO/YES (circle one) 

If YES, what simulations have you used, and what skills were they used to train (for example, 
first-person shooter, flight simulator, operational tactics, etc.)?_ 


2. Do you have experience playing commercial video games? 

NO/YES (circle one) 

If YES, what games have you played, and what kind of games were they (for example, first- 
person shooter, flight simulation, sports, fantasy, etc.)?_ 


If YES, how often do you use video games? 

Approximately_hours per day / week / month / year (circle one) 

3. Do you own a personal computer? 

NO/YES (circle one) 

4. How often do you use a computer off duty ? 

Approximately_hours per day / week / month / year (circle one) 

5. How often do you use a computer on duty ? 

Approximately_hours per day / week / month / year (circle one) 

6. What hand do you use to operate a computer mouse? 

LEFT / RIGHT / EITHER (circle one) 

7. How often do you use the following computer applications? (please answer all that apply) 

a. E-mail: Approximately_hours per day/week/month/year (circle one) 

b. Browse web: Approximately_hours per day / week / month / year (circle one) 

c. Video: Approximately_hours per day/week/month/year (circle one) 

d. Music: Approximately_hours per day/week/month/year (circle one) 

e. Word: Approximately_hours per day/week/month/year (circle one) 

Processor 

f. Other: Approximately_hours per day/week/month/year (circle one) 

8. Year of birth: _ 

9. Primary MOS or job specialty: _ 

10. Rank: _Service: _ 

11. Time in service: _years_months ** TURN PAPER OVER 
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12. How many times have you deployed to a combat theater?_ 

13. If you deployed to a combat theater, did you ever participate in convoy operations as a driver, assistant 
driver, or vehicle gunner? 

NO/YES (circle one) 

If YES, approximately how many?_ 

14. How long have you served with your current unit? _years_months 

16. Rate your agreement with the following statements by marking an X in one block for each: 

DISAGREE AGREE 



1 

2 

3 

4 

5 

Computer-based simulation is an effective training tool. 






Today’s planned computer-based simulation training will 
improve my ability to conduct convoy operations. 






Today’s planned computer-based simulation training will 
improve mv unit’s ability to conduct convoy operations. 






1 think a unit should use computer-based simulation in its 
tactical training. 







17. Write the number of times you have used a sand table or computer-based simulation for each of the 
following and mark your preference: ___ 



# Sand Table 

# Simulation 

Preference (circle one) 

Training 



Sand table 

/ Simulation 

Mission Planning 



Sand table 

/ Simulation 

Mission Briefing 



Sand table 

/ Simulation 

Mission Rehearsal 



Sand table 

/ Simulation 


18. Rate your knowledge of the following small unit convoy tactics, techniques, and procedures by marking 
an X in one block for each skill: 


NOT AT ALL VERY 

PROFICIENT PROFICIENT 



1 

2 

3 

4 

5 

React to an unexploded Improvised Explosive Device 






React to an Improvised Explosive Device detonation 






Take immediate action against a blocked ambush 






Take immediate action against an unblocked ambush 






Cordon and 360 degree security 






Employ vehicle machine guns / weapons 






Mounted fire and maneuver 






Shift fires / cease fires 






Vehicle recovery / bump plan 






Casualty evacuation 






Communication with higher headquarters 






Communication between vehicles in convoy 






Communication between personnel in vehicle 
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APPENDIX M. DEMOGRAPHIC SURVEY (EXPERIMENT 4) 


Please fill in the following questionnaire. All information will be held confidential. 

1. Were you required to use computer-based simulations as part of your training in the past? 

NO/YES (circle one) 

If YES, what simulations have you used, and what skills were they used to train (for example, 
first-person shooter, flight simulator, operational tactics, etc.)?_ 


2. Do you have experience playing commercial video games? 

NO/YES (circle one) 

If YES, what games have you played, and what kind of games were they (for example, first- 
person shooter, flight simulation, sports, fantasy, etc.)?_ 


If YES, how often do you use video games? 

Approximately_hours per day / week / month / year (circle one) 

3. Do you own a personal computer? 

NO/YES (circle one) 

4. How often do you use a computer off duty ? 

Approximately_hours per day / week / month / year (circle one) 

5. How often do you use a computer on duty ? 

Approximately_hours per day / week / month / year (circle one) 

6. What hand do you use to operate a computer mouse? 

LEFT / RIGHT / EITHER (circle one) 

7. How often do you use the following computer applications? (please answer all that apply) 

a. E-mail: Approximately_hours per day/week/month/year (circle one) 

b. Browse web: Approximately_hours per day / week / month / year (circle one) 

c. Video: Approximately_hours per day/week/month/year (circle one) 

d. Music: Approximately_hours per day/week/month/year (circle one) 

e. Word: Approximately_hours per day/week/month/year (circle one) 

Processor 

f. Other: Approximately_hours per day/week/month/year (circle one) 

8. Year of birth: _ 

9. Primary MOS or job specialty: _ 

10. Rank: _Service: _ 

11. Time in service: _years_months ** TURN PAPER OVER 
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12. How many times have you deployed to a combat theater?_ 

13. If you deployed to a combat theater, did you ever participate in convoy operations as a driver, 
assistant driver, or vehicle gunner? 

NO/YES (circle one) 

If YES, approximately how many?_ 

14. How long have you served with your current unit? _years_months 

16. Rate your agreement with the following statements by marking an X in one block for each: 


DISAGREE AGREE 



1 

2 

3 

4 

5 

Computer-based simulation is an effective training tool. 






Today’s planned computer-based simulation training will 
improve my ability to conduct convoy operations. 






Today’s planned computer-based simulation training will 
improve mv unit’s ability to conduct convoy operations. 






1 think a unit should use computer-based simulation in its 
tactical training. 







17. Write the number of times you have used a sand table or computer-based simulation for 
each of the following and mark your preference: __ 



# Sand Table 

# Simulation 

Preference (circle one) 

Training 



Sand table 

/ Simulation 

Mission Planning 



Sand table 

/ Simulation 

Mission Briefing 



Sand table 

/ Simulation 

Mission Rehearsal 



Sand table 

/ Simulation 
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18. Rate your knowledge of the following small unit convoy tactics, techniques, and procedures 
by marking an X in one block for each skill: 

NOT AT ALL VERY 

PROFICIENT PROFICIENT 



1 

2 

3 

4 

5 

React to an unexploded Improvised Explosive Device 






React to an Improvised Explosive Device detonation 






Take immediate action against a blocked ambush 






Take immediate action against an unblocked ambush 






Cordon and 360 degree security 






Employ vehicle machine guns / weapons 






Mounted fire and maneuver 






Shift fires / cease fires 






Vehicle recovery / bump plan 






Casualty evacuation 






Communication with higher headquarters 






Communication between vehicles in convoy 






Communication between personnel in vehicle 







19. Rate your unit’s knowledge of the following small unit convoy tactics, techniques, and 
procedures by marking an X in one block for each skill: 


NOT AT ALL VERY 

EFFECTIVE EFFECTIVE 



1 

2 

3 

4 

5 

React to an unexploded Improvised Explosive Device 






React to an Improvised Explosive Device detonation 






Take immediate action against a blocked ambush 






Take immediate action against an unblocked ambush 






Cordon and 360 degree security 






Employ vehicle machine guns / weapons 






Mounted fire and maneuver 






Shift fires / cease fires 






Vehicle recovery / bump plan 






Casualty evacuation 






Communication with higher headquarters 






Communication between vehicles in convoy 






Communication between personnel in vehicle 







231 



THIS PAGE INTENTIONALLY LEFT BLANK 


232 



APPENDIX N. POST-TRAINING SURVEY (EXPERIMENTS 2 - 4) 


1. Circle your role in the evaluation exercise: DRIVER GUNNER OTHER (circle one) 

If you were not a driver or gunner, what was your role?_ 

2. Rate your agreement with the following statements by marking an X in one block for each: 


DISAGREE AGREE 



1 

2 

3 

4 

5 

Computer-based simulation is an effective training tool. 






Today’s computer-based simulation training improved rriy 
ability to conduct small unit tactical convoy operations. 






Today’s computer-based simulation training improved rriy 
unit’s abilitv to conduct small unit tactical convov 
operations. 






I think a unit should use computer-based simulation in its 
tactical training. 







3. If you had to choose a single training medium, would you prefer to prepare for a small unit 
tactical convoy using a sand table or a computer-based simulation? 

SAND TABLE / COMPUTER SIMULATION (circle one) 


4. Rate your knowledge of the following small unit convoy tactics, techniques, and procedures by 
marking an X in one block for each: 


NOT AT ALL VERY 

PROFICIENT PROFICIENT 



1 

2 

3 

4 

5 

React to an unexploded Improvised Explosive Device 






React to an Improvised Explosive Device detonation 






Take immediate action against a blocked ambush 






Take immediate action against an unblocked ambush 






Cordon and 360 degree security 






Employ vehicle machine guns / weapons 






Mounted fire and maneuver 






Shift fires / cease fires 






Vehicle recovery / bump plan 






Casualty evacuation 






Communication with higher headquarters 






Communication between vehicles in convoy 






Communication between personnel in vehicle 







233 



5. Rate the effectiveness of this computer simuiation exercise in improving your unit’s 
performance by marking an X in one biock for each skiii: 


NOT AT ALL VERY 

EFFECTIVE EFFECTIVE 



1 

2 

3 

4 

5 

React to an unexploded Improvised Explosive Device 






React to an Improvised Explosive Device detonation 






Take immediate action against a blocked ambush 






Take immediate action against an unblocked ambush 






Cordon and 360 degree security 






Employ vehicle machine guns / weapons 






Mounted fire and maneuver 






Shift fires / cease fires 






Vehicle recovery / bump plan 






Casualty evacuation 






Communication with higher headquarters 






Communication between vehicles in convoy 






Communication between personnel in vehicle 







6. Rate your agreement with the foliowing statements by marking an X in one block for each: 

DISAGREE AGREE 



1 

2 

3 

4 

5 

This training mission was successful. 






During this exercise, I felt like my actions in the virtual 
environment had no consequences. 






During this exercise, I felt like I was playing a game. 






During this exercise, I felt like I was conducting training. 






During this exercise, I felt like I was part of the group 
working together. 






During this exercise, I felt isolated from the others. 






This computer simulation provided sufficient audio cues 
for me to know what was going on. 






This computer simulation provided sufficient visual cues 
for me to know what was going on. 






The training value of this exercise came from the 
debriefing and not the exercise itself. 
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7. What task was hardest for you? 


8. What task was easiest for you? 


9. What task was hardest for your vehicle crew? 


10. What task was easiest for your vehicle crew? 


11. What task was hardest for your unit? 


12. What task was easiest for your unit? 


13. How did this simulation exercise help you or your unit? 


14. How did this simulation exercise waste time for you or your unit? 


15. Do you have any suggestions for improving the convoy training? 


16. Provide any other comments or suggestions that you may have. 

Commenti: _ 


Commenti: 


Commenti: 


Commenti: 


Commenti: 


Thank you for volunteering to participate in this study. 
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APPENDIX O. POST-TRAINING SURVEY (EXPERIMENT 4 

CONTROL) 


1. Circle your role in the exercise: DRIVER GUNNER VEHICLE COMMANDER (circle one) 

If you were a unit commander, what was your billet?_ 

2. Rate your knowledge of the following small unit convoy tactics, techniques, and procedures by marking 
an X in one block for each: 


NOT AT ALL VERY 

PROFICIENT PROFICIENT 



1 

2 

3 

4 

5 

React to an unexploded Improvised Explosive Device 






React to an Improvised Explosive Device detonation 






Take immediate action against a blocked ambush 






Take immediate action against an unblocked ambush 






Cordon and 360 degree security 






Employ vehicle machine guns / weapons 






Mounted fire and maneuver 






Shift fires / cease fires 






Vehicle recovery / bump plan 






Casualty evacuation 






Communication with higher headguarters 






Communication between vehicles in convoy 






Communication between personnel in vehicle 







3. Rate the effectiveness of this training exercise in improving your unit’s performance by marking an X in 
one block for each skill: 


NOT AT ALL VERY 

EFFECTIVE EFFECTIVE 



1 

2 

3 

4 

5 

React to an unexploded Improvised Explosive Device 






React to an Improvised Explosive Device detonation 






Take immediate action against a blocked ambush 






Take immediate action against an unblocked ambush 






Cordon and 360 degree security 






Employ vehicle machine guns / weapons 






Mounted fire and maneuver 






Shift fires / cease fires 






Vehicle recovery / bump plan 






Casualty evacuation 






Communication with higher headquarters 






Communication between vehicles in convoy 






Communication between personnel in vehicle 
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4. What task was hardest for you? 


5. What task was easiest for you? 


6. What task was hardest for your vehicle crew? 


7. What task was easiest for your vehicle crew? 


8. What task was hardest for your unit? 


9. What task was easiest for your unit? 


10. How did this training exercise help you or your unit? 


11. How did this training exercise waste time for you or your unit? 


12. Do you have any suggestions for improving the convoy training? 


13. Provide any other comments or suggestions that you may have. 

Commenti: _ 


Commenti: 


Commenti: 


Commenti: 


Commenti: 


Thank you for volunteering to participate in this study. 
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APPENDIX P. KNOWLEDGE TEST 1 (EXPERIMENTS 3 - 4) 


1. B For an unblocked ambush, a vehicle in the kill zone should 

a. Find cover 

b. Continue to move 

c. Stop 

d. Assault 

2. B Which of the following is not a difference between short and long security halts? 

a. Amount of dismounted support employed 

b. Manning crew vehicle crew served weapons 

c. Amount of cover employed by dismounts 

d. Vulnerability of main body 

3. D Which action sequence best describes reaction to a sniper? 

a. Security element provides support by fire while main body presses through the kill 
zone 

b. Security element provides support by fire while main body assaults 

c. Main body provides support by fire while security element assaults 

d. Vehicles return suppressive fire, speed up, and keep moving 

4. C Who is an appropriate person to conduct lED reporting? 

a. Convoy commander 

b. Gunner of the vehicle who spotted the lED 

c. Assistant convoy commander 

d. Security element commander 

5. A Which part of the convoy typically handles navigation? 

a. Head 

b. Security 

c. Combat Assault 

d. Main body 

6. A When analyzing the mission, the convoy commander should follow which process? 

a. Mission, Enemy, Terrain and weather. Troops and fire support. Time (METT-T) 

b. Obstacles, Key terrain. Observation and fields of fire. Cover and concealment. 
Avenues of approach (OKOCA) 

c. Defend, Reinforce, Assault, Withdraw, Delay (DRAW-D) 

d. Confirm, Clear, Cordon, Check, Control (5 C’s) 

7. D The last step in reacting to an lED is: 

a. Cordon 

b. Report the lED to higher headquarters 

c. Consolidate 

d. Control 

8. D Where in the convoy should the convoy commander place himself? 

a. Lead vehicle 

b. Trail vehicle 

c. With the security element commander 

d. Main body 

9. B Which is one of the three elements of a convoy task organization? 

a. Trail 

b. Security 

c. Main body 

d. Head 
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10. A Which is the first step in the recovery of a disabied vehicie? 

a. Estabiish iocai security at the vehicie site 

b. Report to higher headquarters 

c. Convoy continues past the vehicie 

d. Nearest vehicie rigs for tow with strap, chain, cabie, or tow bar 


11. B in order to maintain fiexibiiity in the face of a potentiai threat, a convoy commander 
shouid: 


a. Focus on speed and momentum throughout the convoy 

b. Pian an aiternate route 

c. Piace himseif in the iead of the convoy 

d. Piace himseif in the traii of the convoy 


12. C Who supervises recovery operations for the convoy? 

a. Vehicie commander 

b. Convoy commander 

c. Assistant convoy commander 

d. Security force commander 

13. D Who controis the empioyment of direct and indirect fire assets for the convoy? 

a. Vehicie commander 

b. Convoy commander 

c. Assistant convoy commander 

d. Security force commander 


14. C Which is not one of the “5 C’s”? 

a. Cordon 

b. Controi 

c. Cover 

d. Ciear 


15. A When a convoy reacts to an unbiocked ambush, the most criticai information to pass to 
the convoy commander is: 

a. Direction of enemy fires 

b. Speed of the iead vehicie 

c. Nearest checkpoint to the iead vehicie 

d. Number of vehicies taking fire 

16. A Which is an appropriate range for the vehicie intervai in generai open terrain? 

a. 50-100 meters 

b. 20 - 50 meters 

c. 100 - 150 meters 

d. 150 - 200 meters 

17. C When crossing a danger area, what shouid happen after the traii eiement punches up 
from the rear to provide over watch? 

a. Convoy resumes normai speed 

b. Main body crosses the danger area 

c. Head crosses the danger area 

d. Head provides over watch on the near side 

1 8. D For a biocked ambush, a vehicie that has not yet entered the kiii zone shouid 

a. Assauit 

b. Continue to move 

c. Speed up 

d. Find cover 
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APPENDIX Q. KNOWLEDGE TEST 2 (EXPERIMENTS 3 - 4) 


1. C Which of the following should not be done at a short security halt? 

a. Conduct “5 and 25” meter checks 

b. Maintain 360 degree security 

c. Dismounts establish a secure perimeter 

d. Man crew served weapons 

2. B Who maintains communication with higher and adjacent authorities for the convoy? 

a. Vehicle commander 

b. Convoy commander 

c. Assistant convoy commander 

d. Security force commander 

3. C Which is one of the three organizational elements of a convoy? 

a. Security element 

b. Obstacle Clearing Detachment 

c. Trail 

d. Assault force 

4. B When crossing a danger area, which element provides over watch while the main body 
crosses? 

a. Head 

b. Trail 

c. Main body’s security elements 

d. Combat Assault Element 

5. A Which of the following is not a difference in reaction between blocked and unblocked 
ambushes? 

a. Whether escort vehicles maneuver to put supporting fires on the enemy 

b. Whether vehicles in the kill zone get out 

c. Whether trail elements stop and seek cover 

d. Whether an alternate route is considered 

6. B The primary mission of a convoy is: 

a. Route clearance 

b. Moving personnel and / or cargo 

c. Intelligence collection 

d. Destroying enemy combatants 

7. D When reacting to an lED, the convoy commander should follow which process? 

a. Mission, Enemy, Terrain and weather. Troops and fire support. Time (METT-T) 

b. Obstacles, Key terrain. Observation and fields of fire. Cover and concealment. 
Avenues of approach (OKOCA) 

c. Defend, Reinforce, Assault, Withdraw, Delay (DRAW-D) 

d. Confirm, Clear, Call, Cordon, Control (5 C’s) 

8. B Which of the following should not be used to determine convoy speed? 

a. Terrain 

b. Need for crew rest 

c. Weather conditions 

d. Likelihood of enemy contact 

9. A Which action should the convoy commander avoid when reacting to a sniper? 

a. Stop 

b. Speed up 

c. Return fire 

d. Use smoke 
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10. A Higher headquarters maintains situational awareness of the convoy by monitoring the 
following control measure: 

a. Checkpoints 

b. Rally points 

c. Coordination points 

d. Waypoints 

11. D If there is only one combat life saver, where in the convoy should he be placed? 

a. With the convoy commander 

b. In one of the lead vehicles 

c. In one of the main body vehicles 

d. In one of the trail vehicles 

12. C Who is an appropriate person to conduct casualty reporting? 

a. Combat life saver 

b. Corpsman 

c. Assistant convoy commander 

d. Security element commander 

13. D When a convoy vehicle first becomes disabled, the security element commander should 
consider: 

a. Providing local security at the vehicle site 

b. Punching forward to determine whether enemy are waiting to ambush 

c. Ensuring the convoy stops in a herringbone in case of air attack 

d. Providing security for the convoy main body 

14. C For an unblocked ambush, the element that assaults from outside the kill zone is the 

a. Trail 

b. Main body 

c. Security 

d. Armored escort 

15. D The first step in reacting to an lED is: 

e. Cordon 

f. Report the lED to higher headquarters 

g. Call EOD 

h. Confirm 

16. D Which part of the convoy typically sets the pace? 

a. Head 

b. Security 

c. Combat Assault 

d. Main body 

17. A Which is not a responsibility of the vehicle commander? 

a. Observe sectors of fire 

b. Provide direction to the driver 

c. Communicate with the convoy commander 

d. Designate sectors of observation 

18. B It would be inappropriate to speed up for which immediate action drill? 

a. Reaction to sniper 

b. Reaction to unexploded I ED 

c. Reaction to unblocked ambush 

d. All of the above 
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APPENDIX R. OPORD SHELL (EXPERIMENT 4) 


Scenario:Combat Convoy 


Common Skill: Convoy Operations 
Unit Level: Platoon 

Condition: The platoon has received a fragmentary order (FRAGO) to conduct a combat 

patrol through the designated AO in order to deny the enemy the ability to 
interfere with Coalition vehicular traffic in the AO. All necessary personnel and 
equipment are available. The company has been provided guidance on the rules 
of engagement (ROE) and / or rules of interaction (ROI) per unit SOP. 

Standard: The platoon conducts in accordance with tactical standing operating procedures 

(TSOP), the order, and / or higher commander's guidance. The platoon complies 
with the ROE and or ROI. 
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Scenario Mission Order 


OPORD #1 


Copy_of_copies 

XXXX 

Number-1 

MOVEMENT ORDER (1) (COMBAT PATROL) 

BASIC ORDER 

Ref: (a) Map: Provided 

Time Zone : U 


X Bn, X MARINES 
XXXXXXU XXX 

Message Ref 


Task Organization: No change. 

I. SITUATION 

A. Terrain and Weather. 

29 PALMS: Terrain favors the defender. The terrain provides excellent 

observation and fields of fire throughout the zone allowing defending 
forces to mass fires at nearly all locations. The lack of cover and 
concealment at most points enhances this capability. Additionally, 
attacking forces are required to move through severely restrictive 
terrain at several points in the zone. 

SAHRANI: Terrain favors the defender. The restrictive terrain 

canalizes vehicle movement, while high ground provides excellent 
observation and fields of fire. Wooded areas provide good concealment. 
The terrain is suitable for small unit harassing activity. 

1. Obstacles. Canalizing mountain ranges per map provided. 

2. Avenues of approach. See map provided. 

3. Key terrain. See map provided. 

4. Observation and fields of fire. See map provided. 

5. Cover and concealment. See map provided. 

6. Weather. Weather favors the attacker. With clear skies, friendly 
forces have the ability to observe and designate targets throughout the 
zone, thus supporting the employment of artillery, mortars, and fixed- 
and rotary-wing 

close air support. Prevailing winds favor the use of smoke by attacking 
forces. There is little to no precipitation expected. Humidity is 
extremely 

low, and temperatures vary from as low as 40 degrees during the night to 
as 

high as 110 or more during the day. 
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B. Enemy Forces 

1. Overview: Combat patrols throughout the Coalition/Joint Operating 
Area continue to be targeted by insurgents. Administrative and 
logistical constraints continue to force many combat patrols to travel 
along predictable routes though not all routes are experiencing attacks. 
Virtually all combat patrols must be considered possible targets for 
insurgents. Smaller combat patrols are particularly vulnerable. As 
insurgent experience grows, the MARFOR warns of increased frequency, 
efficiency, and sophistication of attacks. At present, roughly 5% of all 
combat patrols are experiencing some type of attack with 27 Coalition 
service members KIA in the last 60 days. Historically, during relief-in¬ 
place operations, attacks have increased in frequency and lethality. 

The insurgents continue to strike primarily at night, using spotters 
with 

cell phones to alert attackers of approaching combat patrols. However, 
in 

recent weeks due to insurgent success they are becoming increasingly 
brazen 

with more attacks occurring during daylight hours. The pace of insurgent 
attacks is likely to increase in the next few months as the Coalition 
prepares to conduct RIP operations. 

2. The enemy composition has been sporadic in the recent past, ranging 
from single shooters to platoon sized elements executing complex 
attacks, integrating direct and indirect fires with victim actuated, 
radio controlled, and hard-wire command detonated lEDs and/or obstacles. 
The enemy also typically uses secondary lEDs in conjunction with a 
primary lED attack lOT target first responders and security forces. The 
enemy typically uses red or bright pink detonation cord to prime shots, 
and white or copper firing wire to detonate hard-wired lEDs. The enemy 
has indirect capability with 82mm (range 3000m) and 120mm mortars (range 
5700m); the mortars have been used primarily for harassing fires but 
have been effectively integrated into 

complex attacks. The enemy has emplaced multiple AT mines along the MSRs 
and 

secondary roads but does not have the engineering capability to lay a 
traditional minefield. The mines have consisted of AT mines; the AT 
mines 

have been used in a double-stack and pressure accumulation 
configurations in 

a 100 m (or less) stretch of road. 

3. Enemy's most probable course of action: 

a. Observation: Enemy forces have the capability to observe coalition 
forces upon crossing LD and will be able to provide advance warning due 
to their spotters using long range cell phones. 

b. Indirect: Enemy fires will consist of harassing fires from 82mm and 
120mm mortars with possible 122mm rocket fire. The enemy will use their 
indirect fires to cover their emplaced mines and obstacles lOT disrupt 
coalition formations and to prevent CF from successfully breaching 
obstacles. 
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c. Direct: Enemy will employ direct fire with a variety of weapons 
including AK47s (max eff range- 200-300m), RPKs (800m), FN MAG 7.62 
(1200m) and RPG-7s (moving tgt-300m/stat tgt-500m). Enemy direct fire 
engagements will range from single shooters to integration of direct 
fires with IDF/IEDs/mines and obstacles in a complex attack/blocked 
ambush. 

d. Obstacles: Enemy obstacles will most likely consist of deliberately 
emplaced lEDs made with varying types of explosives and military 
ordnance. The enemy has been known to initiate lEDs via radio 
controlled, hard wired, and victim actuated methods. Mines are also 
routinely used to harass CF and inflict CF casualties. Two to four mines 
are typically placed on the road and shoulders in a 50 to 100 m stretch 
of road. The enemy in this area typically uses TM57 and/or TM62 AT 
mines. The most common wire used for hard wired lEDs is thin copper wire 
or white lamp cord. Detonation cord is typically red. 

e. Chemical: The enemy has detonated (1) chlorine enhanced lED 
with in the last 30 days. 

f. Air: Enemy has no attack aviation capability. 

g. Reserve: The enemy has little or no capability to employ a reserve 
and will withdraw to another ambush site or attempt to blend in with the 
civilian populace. 

h. Electronic: The enemy has limited EW capability; they have the 
capability of eavesdropping and jamming commercial (i.e. Motorola) 
radios operating on unsecure nets. 

4. Enemy most dangerous course of action: The enemy will conduct a 
complex attack integrating multiple lEDs or mines to disrupt the CF 
combat patrol, employing both indirect and direct fire capabilities to 
piecemeal the CF patrol. 

C. Friendly Forces 

1. Higher: See Annex J 

2. Adjacent: See Annex J 

3. Attachments and Detachments: See Annex A 

4. Legal Considerations: See Annex J 

II. MISSION. On order, X Company, Y Battalion, Z Marines conducts combat 
patrols through the designated AO to destroy enemy forces in zone in 
order to deny the enemy the ability to interfere with Coalition 
vehicular 

traffic in the AO. 

III. EXECUTION 

A. Commander's Intent and Concept of Operation 

1. Commander's Intent. The purpose of this operation is to gain 
contact with enemy forces and destroy them in order to create conditions 


246 



for the enemy that make their efforts to interfere with Coalition 
operations too risky to be worth their while. The method we will use is 
a series of properly coordinated, "hardened" combat patrols designed to 
maximize protection to troops and vehicles; make an unattractive target 
to insurgents; pose as a deterrent force to current and prospective 
insurgents; and, where possible, attrite insurgent forces. The desired 
end state is significant attrition of anti-coalition forces that limits 
their ability to recruit new fighters, and relative freedom of movement 
along key MSRs for Coalition forces. 

2. Concept of the operation: See Annex C (Operations). This operation 
will consist of three phases. Phase I is the assembly area phase and 
consists of vehicle and personnel preparation, the orders process, and 
pre-combat checks and inspections. Phase II is the movement phase and 
consists of the actual combat patrol movement along the designated 
route. Phase III is the reconstitution phase. Phase III is on-order and 
consists of Class III and V re-supply and follow-on movement. 

B. Tasks 

1. Patrol Leader 

a. At DDTTTTU MMM YY, conduct a combat patrol in the designated AO 
lOT deny the enemy the ability to disrupt coalition activities in 
sector. 

b. The convoy will consist of no more than 8 firing vehicles. 

c. Task the Assistant Patrol Leader and subordinate element leaders 
as necessary. 

d. Coordinate movement during Phase II with X Battalion, X Marines 
COC. 

e. Designate command, control, and communications assignments and 
techniques that will be employed to carry out the move. 

f. Properly integrate attachments. 

g. Execute necessary orders, rehearsals, pre-combat checks, and 
inspections. 

h. Complete Convoy After Action Report per SOP. 

2. Assistant Patrol Leader 

a. At DDTTTTU MMM YY, assist in the conduct of a combat patrol in the 
designated AO lOT deny the enemy the ability to disrupt coalition 
activities in sector. 

b. Supervise marshalling activities in the Assembly Area. 

c. Supervise rehearsals, pre-combat checks, and inspections. 

d. Supervise SOP reiteration and enforcement. 

e. Supervise enroute maintenance and accountability. 
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C. Coordinating Instructions: See Annex C Appendix 18 (Operations 
Overlay) 

1. Rank X. X. XXXXXX is designated convoy commander. 

2. Designated elements OPCON to convoy commander NLT 24 hours prior to 
execution. 

3. Convoy task organization will terminate on order. 

4. MOPP Level 0 in effect. 

5. COMBAT PATROL ASSEMBLY AREA: designated per map provided 

IV. ADMINISTRATION & LOGISTICS 

A. Administration. Additional Control Measures: Used to organize and 
report combat patrol movement and position: 

CP GRID LOCAL DESIGNATION 

See scenario map. 

B. Logistics 

1. MREs issued as required. 

2. Ammunition staged in Assembly Area; issued per combat patrol 
Commander's guidance. 

3. CASEVAC support provided by DET-XX. 

4. Patrols and convoys will tow all damaged vehicles to the release 
point. 

V. COMMAND AND SIGNAL 

A. Command Relationships. Per Battalion SOP. 

B. Command Posts and Headquarters. Located at Camp Wilson. 

C. Signal. Per SOP. 


ACKNOWLEDGE RECEIPT 


R. F. DAGGER 
COL, USMC 
COMMANDING 

ANNEXES: 

None. 

OFFICIAL: 

s/ 

C. J. SABER 
LTCOL, USMC 

OPERATIONS OFFICER 
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APPENDIX S. LIVE CONVOY EVALUATION (EXPERIMENT 4) 


MOTORIZED PATROL CHECKLIST 


CO. 


PET. 


SEC. 


DATE 


TIME 


PATROL LEADER- 


#VICS/PAX 


1. Prior to DFL 

□ WarnO issued?. 
EragO issued? _ 


□ 

□ 

□ 


Einal coordination and intelligence brief. 
PCC/PCCI_ 


6 


7 


□ 

□ 


Mission card submitted 


10 


2. Depart Friendly Lines 

□ Short count_ 


□ 


Comm check, move to AA 

tI 


□ 

□ 

□ 


Dismount & Condition 4. 
Check out w/ COC_ 


10 


3. Patrol Movement 

□ Navigator designated/employed?. 
PL positioned for best C2?_ 


□ 

□ 

□ 


□ 


□ 


□ 


Lead trace reporting for turns or direction changes . 

hide Control Measures_ 


Use of Vel 

2Q 3 


4 


6 


7 


Dispersion and speed appropriate to METT 

2 


4 


6 


7 


Use of checkpoints/HHQ reporting 

2 


4 


6 


10 


10 


10 


4. Immediate Actions 

a. Short Security Halt 
Duration per SOP?. 


□ 

□ 


1 


Eire/ol 

2 


bservation sectors covered? 

3rn 4ri sn 6 


□ 
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1 


1 


1 


1 

1 

1 


b. Long SecuriP^ Halt 
□ Position improvement 


10 


□ Security established 
2 


□ 


10 


c. PIED spotted 
□ Brevity code used? 


2 

□ 

□ 

3 

n 

4 

n 

5 

n 

6 

n 

7 

n 

8 

n 

9 

n 

10 

n 


d. lED found 
□ Confirm 

2 

□ Cl 

2 

□ Cl 

2 

□ 

ord( 

□ 

ontr 

□ 

3 

)n _ 

□ 

4 

□ 

5 

□ 

6 

□ 

7 

□ 

8 

□ 

9 

□ 

10 

□ 


3 

ol _ 

n 

4 

n 

5 

n 

6 

n 

7 

n 

8 

n 

9 

n 

10 

n 


3 

n 

4 

n 

5 

n 

6 

n 

7 

n 

8 

n 

9 

n 

10 

n 



□ Call_ 

□ Clear. 

□ _ 


e. 

□ 

1 


□ 

□ 

□ 

□ 


Vehicle Recovery 
Rehearsal evident? 



5 


6 


Recovery vehicle designated 

Equipment available_ 

Obscuration/deception used? 


7 



10 
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/. Casualty Evacuation 

□ Rehearsal evident?_ 

l | I 2 1 I 3 1 I 4 1 I 5 1 I 6 1 I 7 1 I 8 1 | 9 | | 101 

□ Aid and litter teams or vehicles_ 

□ HHQ reporting_ 

□ Triage and mission impact_ 

l | I 2 P] 3 1 I 4 1 I 5 1 I 6 1 I 7 1 I 8 1 I 9 1 | 101 



3 


4 

□ 

5 

□ 

6 

□ 

7 

□ 

8 

□ 

9 

□ 

10 

□ 


Actions on Contact 

PID of targets 

2 sQ 

Sectors of fir 

4 

e 

□ 

5 

□ 

6 

□ 

7 

□ 

8 

□ 

9 

□ 

10 

□ 


2 \ 

3 


4 

□ 

5 

□ 

6 

□ 

7 

□ 

8 

□ 

9 

□ 

10 

□ 



□ Fire discipline_ 

iQ 3Q 4Q sQ 6Q 7^ 

□ R ehea rse d res po nse? 

l | I 2 1 I 3 1 I 4 1 I 5 1 I 6 1 I 7 1 

□ Individual actions support PL SOM_ 

l | I 2 1 I 3 1 I 4 1 I 5 1 I 6 1 I 7 1 

□ Vehicle-dismount integration_ 

l | I 2 1 I 3 1 I 4 1 I 5 1 I 6 1 I 7 1 

□ C SW fir e sup po rt shi fte d and /o r cea se d 

l | I 2 1 I 3 1 I 4 1 I 5 1 I 6 1 I 7 1 

□ C onso lid ation o n obj ect ive 

l | I 2 1 I 3 1 I 4 1 5 1 I 6 1 I 7 1 

□ Pursuit and continuing actions_ 

l | I 2 1 I 3 1 I 4 1 I 5 1 I 6 1 I 7 1 

1. Reporting to HHQ 

2. Casualties 

3. Detainees 


8 1 I 9 1 I 101 

8 1 I 9 1 I 101 

8 1 I 9 1 I 101 

8 1 I 9 1 I 101 

8 1 I 9 1 I 101 

8 1 I 9 1 I 101 

8 1 I 9 1 I 101 


4. Impact on civil populace 

5. Accountability 

6. Redistribution/recovery 
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5 . 


Vehicle Control Points 

a. Snap VCP 



1 


1 


□ Proper vehicle search_ 

□ Proper personnel search_ 

□ Use of interpreter or HN forces 

□ _ 


b. 

Hasty VCP 





□ 

Long halt security 





□ 

Rehearsal evident? 





□ 

Barrier plan effective 





□ 

Fields of fire acceptable 






2 3 4 5 6^ 


] c 


] 10 

□ 

Equipment/signage employed 





□ 

Trigger points established 





□ 

Force continuum understood_ 






2 3 4 5 6^ 

7 

8 

9 

10 


□ HHQ reporting_ 

□ Personnel search techniques_ 

□ Vehicle search techniques_ 

□ Intelligence collection_ 

□ Use of interpreter or HN forces 

□ Basic Arabic commands used? 

□ _ 


c. Deliberate VCP 

i. Not employed 


6. Re-entering Friendly Lines 

□ Request made before reaching gate _ 

□ VIC/PAX count modified, if needed 

□ Dismount and go Condition 4_ 

□ Conduct AAR_ 

□ Detainee procedures_ 

□ Prepare for next patrol_ 


Additional Comments: 
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APPENDIX T. STUDENT DEMOGRAPHIC SURVEY 

(EXPERIMENT 5) 


Please fill in the following questionnaire. All information will be held confidential. 

1. Were you required to use computer-based simulations as part of your training in the past? 

NO / YES (circle one) 

If YES, what simulations have you used, and what skills were they used to train (for example, first- 
person shooter, flight simulator, operational tactics, etc.)?_ 


2. Do you have experience playing commercial video games? 

NO / YES (circle one) 

If YES, what games have you played, and what kind of games were they (for example, first-person 
shooter, flight simulation, sports, fantasy, etc.)?_ 


If YES, how often do you use video games? 

Approximately_hours per day / week / month / year (circle one) 

3. Do you own a personal computer? 

NO / YES (circle one) 

4. How often do you use a computer off duty ? 

Approximately_hours per day / week / month / year (circle one) 

5. How often do you use a computer on duty ? 

Approximately_hours per day / week / month / year (circle one) 

6. What hand do you use to operate a computer mouse? 

LEFT / RIGHT / EITHER (circle one) 


7. How often do you use the following computer applications? (please answer all that apply) 


a. E-mail: Approximately. 

b. Browse web: Approximately 


c. Video: 

d. Music: 

e. Word: 
Processor 

f. Other: 

8. Year of birth: 


Approximately. 

Approximately. 

Approximately. 

Approximately. 


9. Primary MOS or job specialty: 

10. Rank: _ 


hours per day / week / month / year (circle one) 
hours per day / week / month / year (circle one) 
hours per day / week / month / year (circle one) 
hours per day / week / month / year (circle one) 
hours per day / week / month / year (circle one) 

hours per day / week / month / year (circle one) 


11. Time in service: 


.years. 


_Service: 

months 


TURN PAPER OVER 
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12. How many times have you deployed to a combat theater?_ 

13. If you deployed to a combat theater, did you ever participate in convoy operations as a driver, 
assistant driver, or vehicle gunner? 

NO/YES (circle one) 

If YES, approximately how many?_ 

14. How long have you served with your current unit? _years_months 

16. Rate your agreement with the following statements by marking an X in one block for each: 


DISAGREE AGREE 



1 

2 

3 

4 

5 

Computer-based simulation is an effective training tool. 






Today’s planned computer-based simulation training will 
improve my ability to conduct convoy operations. 






Today’s planned computer-based simulation training will 
improve mv unit’s ability to conduct convoy operations. 






1 think a unit should use computer-based simulation in its 
tactical training. 







17. Write the number of times you have used a sand table or computer-based simulation for 
each of the following and mark your preference: __ 



# Sand Table 

# Simulation 

Preference (circle one) 

Training 



Sand table 

/ Simulation 

Mission Planning 



Sand table 

/ Simulation 

Mission Briefing 



Sand table 

/ Simulation 

Mission Rehearsal 



Sand table 

/ Simulation 
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18. Rate your knowledge of the following small unit convoy tactics, techniques, and procedures by marking an X in 
one block for each skill: 


NOT AT ALL VERY 

PROFICIENT PROFICIENT 



1 

2 

3 

4 

5 

React to an unexploded Improvised Explosive Device 






React to an Improvised Explosive Device detonation 






Take immediate action against a blocked ambush 






Take immediate action against an unblocked ambush 






Cordon and 360 degree security 






Employ vehicle machine guns / weapons 






Mounted fire and maneuver 






Shift fires / cease fires 






Vehicle recovery / bump plan 






Casualty evacuation 






Communication with higher headquarters 






Communication between vehicles in convoy 






Communication between personnel in vehicle 






Execute the troop leading steps using BAMCIS 






Conduct mission analysis using METT-T 






Receive a 5 paragraph operational order 






Give a 5 paragraph operational order 







19. Rate your unit’s knowledge of the following small unit convoy tactics, techniques, and procedures by marking 
an X in one block for each skill: 


NOT AT ALL VERY 

EFFECTIVE EFFECTIVE 



1 

2 

3 

4 

5 

React to an unexploded Improvised Explosive Device 






React to an Improvised Explosive Device detonation 






Take immediate action against a blocked ambush 






Take immediate action against an unblocked ambush 






Cordon and 360 degree security 






Employ vehicle machine guns / weapons 






Mounted fire and maneuver 






Shift fires / cease fires 






Vehicle recovery / bump plan 






Casualty evacuation 






Communication with higher headquarters 






Communication between vehicles in convoy 






Communication between personnel in vehicle 
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APPENDIX U. INSTRUCTOR DEMOGRAPHIC SURVEY 

(EXPERIMENT 5) 


Please fill in the following questionnaire. All information will be held confidential. 

1. Were you required to use computer-based simulations as part of your training in the past? 

NO / YES (circle one) 

If YES, what simulations have you used, and what skills were they used to train (for example, first- 
person shooter, flight simulator, operational tactics, etc.)?_ 


2. Do you have experience playing commercial video games? 

NO / YES (circle one) 

If YES, what games have you played, and what kind of games were they (for example, first-person 
shooter, flight simulation, sports, fantasy, etc.)?_ 


If YES, how often do you use video games? 

Approximately_hours per day / week / month / year (circle one) 

3. Do you own a personal computer? 

NO / YES (circle one) 

4. How often do you use a computer off duty ? 

Approximately_hours per day / week / month / year (circle one) 

5. How often do you use a computer on duty ? 

Approximately_hours per day / week / month / year (circle one) 

6. What hand do you use to operate a computer mouse? 

LEFT / RIGHT / EITHER (circle one) 


7. How often do you use the following computer applications? (please answer all that apply) 


a. E-mail: Approximately. 

b. Browse web: Approximately 


c. Video: 

d. Music: 

e. Word: 
Processor 

f. Other: 

8. Year of birth: 


Approximately. 

Approximately. 

Approximately. 

Approximately. 


9. Primary MOS or job specialty: 

10. Rank: _ 


hours per day / week / month / year (circle one) 
hours per day / week / month / year (circle one) 
hours per day / week / month / year (circle one) 
hours per day / week / month / year (circle one) 
hours per day / week / month / year (circle one) 

hours per day / week / month / year (circle one) 


11. Time in service: 


.years. 


_Service: 

months 


TURN PAPER OVER 
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12. How many times have you deployed to a combat theater?_ 

13. If you deployed to a combat theater, did you ever participate in convoy operations as a driver, 
assistant driver, or vehicle gunner? 

NO/YES (circle one) 

If YES, approximately how many?_ 

14. How long have you served with your current unit? _years_months 

16. Rate your agreement with the following statements by marking an X in one block for each: 


DISAGREE AGREE 



1 

2 

3 

4 

5 

Computer-based simulation is an effective training tool. 






Computer-based simulation training will enable me to 
train my Marines better than if 1 did not have it. 






1 think a unit should use computer-based simulation in its 
tactical training. 







17. Write the number of times you have used a sand table or computer-based simulation for 
each of the following and mark your preference: __ 



# Sand Table 

# Simulation 

Preference (circle one) 

Training 



Sand table 

/ Simulation 

Mission Planning 



Sand table 

/ Simulation 

Mission Briefing 



Sand table 

/ Simulation 

Mission Rehearsal 



Sand table 

/ Simulation 
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18. Rate how effective you anticipate VBS 2™ to be in training each of the following small unit convoy 
tactics, techniques, and procedures by marking an X in one block for each skill: 

NOT AT ALL VERY 

EFFECTIVE EFFECTIVE 



1 

2 

3 

4 

5 

React to an unexploded Improvised Explosive Device 






React to an Improvised Explosive Device detonation 






Take immediate action against a blocked ambush 






Take immediate action against an unblocked ambush 






Cordon and 360 degree security 






Employ vehicle machine guns / weapons 






Mounted fire and maneuver 






Shift fires / cease fires 






Vehicle recovery / bump plan 






Casualty evacuation 






Communication with higher headquarters 






Communication between vehicles in convoy 






Communication between personnel in vehicle 






Execute the troop leading steps using BAMCIS 






Conduct mission analysis using METT-T 






Receive a 5 paragraph operational order 






Give a 5 paragraph operational order 







19. Rate your proficiency for each of the following trainer skills with VBS 2™: 

NOT AT ALL VERY 

PROFICIENT PROFICICENT 


Set up the network and computer hardware 

1 

2 

3 

4 

5 

Set up VBS 2™ for Administrator and User 






Start a scenario that has already been built 






Add entities to a scenario 






Use triggers 






Control civilians and traffic 






Control enemy 






Maintain your situational awareness as the trainer 






Set up CNR Sim and CNR Log 






Use an AAR to facilitate a debrief 






Answer user’s questions about how to use VBS 2™ 






Facilitate a call for fire 






Use the simulation to meet your training objectives 
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20. As a trainer, what training objective or skill do you think VBS 2™ will help you the most to 
train your Marines?_ 


21. As a trainer, what training objective or skill do you think VBS 2™ will help you the least to 
train your Marines?_ 


22. How would you improve this train-the-trainer instructional package?. 
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APPENDIX V. INSTRUCTOR TRAINING NOTES (EXPERIMENT 5) 


♦♦♦ Setting up the DVTE suite’s hardware 

> Setting up computers 

> Connecting the LAN 

❖ Disseminate appropriate files to all user computers 

> Administrator 

■ Go to Windows Explorer 

■ Click on appropriate file 

■ Go to File-Properties and then the “Sharing” tab 

■ Under “Network sharing and security” check the “Share this folder on the network” 
box 

> User 

■ Go to Tools-Map Network Drive, click browse 

■ The shared folder should show up; click on it, and click finish 

❖ Learning to be a VBS 2 user 

> Introduction brief 

> Cheat Sheet 

> Familiarity scenario 

> Training scenarios 

> Library 
CNR 

> Documentation and access through All Programs^Bohemia Interactive^CNR 

> Setting up CNR Log 

■ Configuration-Remote Access^Check “Remote Access Enabled” 

■ Ensure Access Port is 8080 

■ Leave open 

> Set up VBS LVC 

■ Open Notepad 

■ Open c:\Program Files\Bohemia lnteractive\VBS2 VTK\config\VBSCIient.config 

■ Get computer IP address 

• Control Panel-Network Connections-Local Area Connection-Support tab 

■ Change IP address in VBSClient.config to server computer 

> Start CNR Sim on client computers 

■ Set up channels, frequencies, and hot keys 

> Start server computer VBS 2 using batch file in source directory 

■ c:\Program Files\Bohemia lnteractive\VBS2 VTK\VBS2 LVC Game CNR Log.bat 

> Start client computers in normal VBS 2 

> CNR Log must be on for AAR function to work 

❖ Show location of manuals 

> Go to c:\Program Files\Bohemia lnteractive\VBS2 VTK\docs 

❖ Run Training Wheels 

> Use of overhead view 

❖ Setting up the options 

> File is My Documents\VBS2\Administrator VBS2PROFILE file 

> Video Options 

■ Terrain Distance: change to 10,000 

■ Resolution: 1280x 1024x32 

■ Advanced tab: Object Distance: 10,000 

> Controls 

■ Up: Delete “Page Up” 

■ Go Prone: Delete “Page Down” 

■ Map and Hide Map: Delete “M” and replace with “Right CTRL M” 
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- GPS and GPS Toggle: “M” and “2 x M” 

- Voice over net and Push to talk: Delete “Caps Lock” 

■ Talk on Vehicle channel: “Caps Lock” 

Building a scenario 

> Entering as Administrator (Administrator, DVTEM90build2) 

> “File” options 

> “View” options 

> Map manipulation: zoom and slide 

> Adding a unit 

■ Add a server, civilian or blue 

■ Naming, Rank, Make Player 

■ Review your work 

■ Add an enemy player 

■ Moving and turning units 

■ Deleting a unit 

> Add a group 

■ Add an enemy fire team 

> Add a vehicle 

■ Unmanned Vehicles—MQ-1 Predator (USAF) 

> Add an empty vehicle 

■ Give yourself a HMMWV 

> Measure distance tool 

■ Add a measure distance point, right click, new distance 

> Add a waypoint 

■ Add a waypoint, and then add more to make a circular path 

■ For the last waypoint in the circle, cross 

• Edit the waypoint and change “MOVE” to “CYCLE” 

> Add a vehicle^Ground control station 

> Add a control link 

■ Right click and “Link to vehicle” and then click on Ground control station to link 

■ Player walks up to the ground control station and treats like vehicle (“U” to interact, 
then scroll to UAV controller) 

> Add a marker (useful for 2D view) 

> Intel: change the weather and time 

> Add an I ED 

> Add a trigger 

■ lED 

■ Artillery strike 

■ Unit / vehicle movement 

Use of AAR tool to debrief an exercise scenario 
Serving as controller for an exercise scenario 

> Artillery 

> Civilians 

> T raff ic 

> Use of triggers to cause casualties and vehicle breakdown 
Use of indirect fire console 

> Visibility settings 

> Putting the console in a mission 

> Les 

> Running the console 

■ Different mission types 

■ Initial call for fire 

■ Adjustments 

■ Fire for effect 

> Using the console in a mission 
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APPENDIX W. STUDENT POST TRAINING SURVEY 

(EXPERIMENT 5) 


1. Circle your role in the exercise: DRIVER GUNNER VEHICLE COMMANDER (circle one) 

If you were a unit commander, what was your billet?_ 

2. Rate your agreement with the following statements by marking an X in one block for each: 


DISAGREE AGREE 



1 

2 

3 

4 

5 

Computer-based simulation is an effective training tool. 






Today’s computer-based simulation training improved my ability 
to conduct small unit tactical convoy operations. 






Todav’s comouter-based simulation trainina imoroved mv unit’s 
ability to conduct small unit tactical convoy operations. 






I think a unit should use computer-based simulation in its 
tactical training. 







3. If you had to choose a single training medium, would you prefer to prepare for a small unit tactical convoy 
using a sand table or a computer-based simulation? 


SAND TABLE / COMPUTER SIMULATION (circle one) 

4. Rate your knowledge of the following small unit convoy tactics, techniques, and procedures by marking 
an X in one block for each: 


NOT AT ALL VERY 

PROFICIENT PROFICIENT 



1 

2 

3 

4 

5 

React to an unexploded Improvised Explosive Device 






React to an Improvised Explosive Device detonatien 






Take immediate action against a blocked ambush 






Take immediate action against an unblocked ambush 






Cordon and 360 degree security 






Empicy vehicle machine guns / weapons 






Mounted fire and maneuver 






Shift fires / cease fires 






Vehicle recovery / bump plan 






Casualty evacuation 






Communication with higher headquarters 






Communicatien between vehicles in convoy 






Communication between personnel in vehicle 






Execute the troop leading steps using BAMCIS 






Conduct mission analysis using METT-T 






Receive a 5 paragraph operational order 






Give a 5 paragraph eperational erder 
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5. Rate the effectiveness of this computer simuiation exercise in improving your unit’s 
performance by marking an X in one biock for each skiii: 


NOT AT ALL VERY 

EFFECTIVE EFFECTIVE 



1 

2 

3 

4 

5 

React to an unexploded Improvised Explosive Device 






React to an Improvised Explosive Device detonation 






Take immediate action against a blocked ambush 






Take immediate action against an unblocked ambush 






Cordon and 360 degree security 






Employ vehicle machine guns / weapons 






Mounted fire and maneuver 






Shift fires / cease fires 






Vehicle recovery / bump plan 






Casualty evacuation 






Communication with higher headquarters 






Communication between vehicles in convoy 






Communication between personnel in vehicle 







6. Rate your agreement with the foliowing statements by marking an X in one block for each: 

DISAGREE AGREE 



1 

2 

3 

4 

5 

This training mission was successful. 






During this exercise, I felt like my actions in the virtual 
environment had no consequences. 






During this exercise, I felt like I was playing a game. 






During this exercise, I felt like I was conducting training. 






During this exercise, I felt like I was part of the group 
working together. 






During this exercise, I felt isolated from the others. 






This computer simulation provided sufficient audio cues 
for me to know what was going on. 






This computer simulation provided sufficient visual cues 
for me to know what was going on. 






The training value of this exercise came from the 
debriefing and not the exercise itself. 
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7. What task was hardest for you? 


8. What task was easiest for you? 


9. What task was hardest for your vehicle crew? 


10. What task was easiest for your vehicle crew? 


11. What task was hardest for your unit? 


12. What task was easiest for your unit? 


13. How did this simulation exercise help you or your unit? 


14. How did this simulation exercise waste time for you or your unit? 


15. Do you have any suggestions for improving the convoy training? 


16. Provide any other comments or suggestions that you may have. 

Commenti: _ 


Commenti: 


Commenti: 


Commenti: 


Commenti: 


Thank you for volunteering to participate in this study. 
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APPENDIX X. INSTRUCTOR POST TRAINING SURVEY 

(EXPERIMENT 5) 


1. Circle your role in the exercise: CONTROLLER ARTILLERY OBSERVER (circle one) 

2. Rate your agreement with the following statements by marking an X in one block for each: 


DISAGREE AGREE 



1 

2 

3 

4 

5 

Computer-based simulation is an effective training tool. 






Computer-based simulation training enabled me to train my 
Marines better than if I did not have it. 






I think a unit should use computer-based simulation in its 
tactical training. 







3. If you had to choose a single training medium, would you prefer to prepare for a small unit tactical 
convoy using a sand table or a computer-based simulation? 

SAND TABLE / COMPUTER SIMULATION (circle one) 

4. Rate how effective you thought VBS 2™ was in training each of the following small unit convoy 
tactics, techniques, and procedures by marking an X in one block for each skill: 


NOT AT ALL VERY 

EFFECTIVE EFFECTIVE 



1 

2 

3 

4 

5 

React to an unexploded Improvised Explosive Device 






React to an Improvised Explosive Device detonation 






Take immediate action against a blocked ambush 






Take immediate action against an unblocked ambush 






Cordon and 360 degree security 






Employ vehicle machine guns / weapons 






Mounted fire and maneuver 






Shift fires / cease fires 






Vehicle recovery / bump plan 






Casualty evacuation 






Communication with higher headquarters 






Communication between vehicles in convoy 






Communication between personnel in vehicle 






Execute the troop leading steps using BAMCIS 






Conduct mission analysis using METT-T 






Receive a 5 paragraph operational order 






Give a 5 paragraph operational order 
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5. Rate which training piatform you wouid prefer to use to train each of the foiiowing smaii unit 
convoy tactics, techniques, and procedures by marking an X in one biock for each skiii: 


VBS2™ Don’t care VCCT 


React to an unexpioded improvised Expiosive Device 






React to an improvised Expiosive Device detonation 






Take immediate action against a biocked ambush 






Take immediate action against an unbiocked ambush 






Cordon and 360 degree security 






Empioy vehicie machine guns / weapons 






Mounted fire and maneuver 






Shift fires / cease fires 






Vehicie recovery / bump pian 






Casuaity evacuation 






Communication with higher headquarters 






Communication between vehicies in convoy 






Communication between personnei in vehicie 






Execute the troop ieading steps using BAMCiS 






Conduct mission anaiysis using METT-T 






Receive a 5 paragraph operationai order 






Give a 5 paragraph operationai order 
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6. Rate your proficiency for each of the following trainer skills with VBS 2™\ 


NOT AT ALL VERY 

PROFICIENT PROFICIENT 



1 

2 

3 

4 

5 

Set up the network and computer hardware 






Set up VBS 2™ for Administrator and User 






Start a scenario that has already been built 






Add entities to a scenario 






Use triggers 






Control civilians and traffic 






Control enemy 






Maintain your situational awareness as the trainer 






Set up CNR Sim and CNR Log 






Use an AAR to facilitate a debrief 






Answer user’s questions about how to use VBS 2™ 






Facilitate a call for fire 






Use the simulation to meet your training objectives 







7. Rate your agreement with the following statements by marking an X in one block for each: 

DISAGREE AGREE 



1 

2 

3 

4 

5 

This training was successful. 






During this exercise, I felt like training was not real 
enough. 






During this exercise, I felt like we were playing a game. 






During this exercise, I felt like we were conducting 
training. 






This computer simulation provided sufficient audio cues 
for me to know what was going on. 






This computer simulation provided sufficient visual cues 
for me to know what was going on. 






The training value of this exercise came from the 
debriefing and not the exercise itself. 
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8. As a trainer, what training objective or skiil do you think VBS 2™ will help you the most to train 
your Marines?_ 


9. As a trainer, what training objective or skill do you think VBS 2™ will help you the least to train 
your Marines?_ 


10. How did this simulation exercise help you or your unit? 


11. How did this simulation exercise waste time for you or your unit? 


12. Do you have any suggestions for improving the simulation training? 


13. Provide any other comments or suggestions that you may have. 

Comment: _ 


Comment: 


Comment: 


Comment: 


Comment: 


Thank you for volunteering to participate in this study. 
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APPENDIX Y. KNOWLEDGE TEST A (EXPERIMENT 5) 


1. B For an unblocked ambush, a vehicle in the kill zone should 

e. Find cover 

f. Continue to move 

g. Stop 

h. Assault 


lA Procedure 

Unblocked 
ambush—does 
student 

understand basic 
execution? 


2. A Higher headquarters maintains situational awareness of the convoy 

by monitoring the following control measure: 

e. Checkpoints 

f. Rally points 

g. Coordination points 

h. Waypoints 

3. C Which is not one of the “5 C’s”? 

e. Cordon 

f. Control 

g. Cover 

h. Clear 


Mission planning 

Does the student 
understand basic 
control measures? 


Reporting 

Does the student 
understand basic 
reporting 
responsibilities? 


4. A What are the three main components of the Situation paragraph of 

the 5 paragraph operational order? 

a. Terrain and weather; enemy situation, friendly situation 

b. Terrain and weather; enemy situation; commander’s intent 

c. Mission; concept of operations; commander’s intent 

d. Beans; bullets; bandaids 

5. C Which of the following should not be done at a short security halt? 

e. Conduct “5 and 25” meter checks 

f. Maintain 360 degree security 

g. Dismounts establish a secure perimeter 

h. Man crew served weapons 

6. A Which is not a responsibility of the vehicle commander? 

e. Observe sectors of fire 

f. Provide direction to the driver 

g. Communicate with the convoy commander 

h. Designate sectors of observation 

7. C Who is an appropriate person to conduct lED reporting to higher 

headquarters? 

e. Vehicle commander of the trail vehicle 

f. Gunner of the vehicle who spotted the lED 

g. Assistant convoy commander 

h. Security element commander 


Mission planning 

OPORD— 
Understanding of 
basic OPORD 
structure 


lA Procedure 

Hails—does 
student 
understand 
difference betw 
iong / short bait? 


Responsibilities 

Does the student 
understand basic 
crew tasks? 


Responsibilities / 
Division of labor 

ACC handies 
administrative so 
CC can fight the 
unit / direct action 


8. D Which of the following would you consider in the “E” part of METT-T? 

a. Weather and its effects 

b. Terrain and its effects 

c. Composition of friendly forces 

d. Composition of enemy forces 


Mission planning 

METT-T— 
Understanding of 
enemy 

considerations in 
mission pianning 
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9. A Which is the first step in the recovery of a disabled vehicle? 

e. Establish local security at the vehicle site 

f. Report to higher headquarters 

g. Convoy continues past the vehicle 

h. Nearest vehicle rigs for tow with strap, chain, cable, or tow bar 


lA Procedure 

Vehicle recover— 
does student 
understand basic 
execution? 


10. C In the 5 paragraph operational order, in which paragraph would you 

task your subordinate units? 

a. Paragraph 1 

b. Paragraph 2 

c. Paragraph 3 

d. Paragraph 4 

11. A Which part of the convoy typically handles navigation? 

e. Head 

f. Security 

g. Combat Assault 

h. Main body 

12. B Which troop leading step allows you to gain information that you 

need to complete your plan? 

a. Begin planning 

b. Make reconnaissance 

c. Execute the operation 

d. Conduct rehearsals 


Mission planning 

OPORD— 
Understanding of 
basic OPORD 
structure 


Organizational 
structure vs task 
organization 

Does student 

understand 

organizational? 


Mission planning 

BAMCIS— 
Understanding of 
basic troop 

leading steps 


13. D Which action sequence best describes reaction to a sniper? 

e. Security element provides support by fire while main body 
presses through the kill zone 

f. Security element provides support by fire while main body 
assaults 

g. Main body provides support by fire while security element 
assaults 

h. Vehicles return suppressive fire, speed up, and keep moving 

14. C When planning an operation, you should take_of the time 

available for yourself and leave_of the time available for 

your subordinates. 

a. 1/2; 1/2 

b. 2/3; 1/3 

c. 1/3; 2/3 

d. 3/4; 1/4 


lA Procedure 

Sniper—does 
student 

understand basic 
execution? 


Mission planning 

Understanding of 
basic mission 
planning 
fundamentals 


15. C Which is one of the three organizational elements of a convoy? 

e. Security element 

f. Obstacle Clearing Detachment 

g. Trail 

h. Assault force 


Organizational 
structure vs task 
organization 

Does student 
understand the 
difference? 


1 6. C If you received a 5 paragraph operational order from your higher 
headquarters, where would you primarily look to find your mission? 

a. Friendly situation 

b. Commander’s intent 

c. Tasks to subordinate units 

d. Coordinating instructions 


Mission planning 

OPORD— 
Understanding of 
basic OPORD 
structure 
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17. C Who supervises recovery operations for the convoy? 

e. Vehicle commander 

f. Convoy commander 

g. Assistant convoy commander 

h. Security force commander 


Responsibilities / 
Division of labor 

Does student 
understand 
delegation of 
logistic functions? 


18. A 


When analyzing the mission, the convoy commander should 
follow which process? 

e. Mission, Enemy, Terrain and weather. Troops and fire support. 
Time (METT-T) 

f. Obstacles, Key terrain. Observation and fields of fire. Cover and 
concealment. Avenues of approach (OKOCA) 

g. Defend, Reinforce, Assault, Withdraw, Delay (DRAW-D) 

h. Confirm, Clear, Cordon, Check, Control (5 C’s) 


Mission Planning 

Basic 

understanding of 
mission planning 
concepts 


19. C For an unblocked ambush, the element that assaults from outside 
the kill zone is the 

e. Trail 

f. Main body 

g. Security 

h. Armored escort 


lA Procedure 

Ambush—does 
student 

understand basic 
execution? 


20. A When a convoy reacts to an unblocked ambush, the most critical 
information to pass to the convoy commander is: 

e. Direction of enemy fires 

f. Speed of the lead vehicle 

g. Nearest checkpoint to the lead vehicle 

h. Number of vehicles taking fire 


lA Procedure 

Unblocked 
ambush—does 
the student 

understand basic 
execution? 


21. D In what part of the 5 paragraph operational order would you publish 
the time line for the operation? 

a. Friendly situation 

b. Mission statement 

c. Tasks to subordinate units 

d. Coordinating instructions 


Mission planning 

OPORD— 
Understanding of 
basic OPORD 
structure 


22. D The last step in reacting to an lED is: 

i. Cordon 

j. Report the lED to higher headquarters 

k. Consolidate 

l. Control 


lA Procedure 

I ED—does 
student 

understand the 5 
C’s? 


23. B In the 5 paragraph operational order, in which paragraph would you 
find the time you should start your mission? 

a. Paragraph 1 

b. Paragraph 2 

c. Paragraph 4 

d. Paragraph 5 


Mission planning 

OPORD— 
Understanding of 
basic OPORD 
structure 


24. A Which is an appropriate range for the vehicle interval in general 
open terrain? 

e. 50- 100 meters 

f. 20 - 50 meters 

g. 200 - 400 meters 

h. 150 - 250 meters 


Movement 

Does the student 
understand basic 
convoy movement 
principles? 
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APPENDIX Z. KNOWLEDGE TEST B (EXPERIMENT 5) 


1. D For a blocked ambush, a vehicle that has not yet entered the kill 

zone should 

a. Assault 

b. Continue to move 

c. Speed up 

d. Find cover 

2. B The primary mission of a convoy is: 

e. Route clearance 

f. Moving personnel and / or cargo 

g. Intelligence collection 

h. Destroying enemy combatants 

3. D When reacting to an lED, the convoy commander should follow 

which process? 

e. Mission, Enemy, Terrain and weather. Troops and fire support. 
Time (METT-T) 

f. 5/25’s 

g. Defend, Reinforce, Assault, Withdraw, Delay (DRAW-D) 

h. Confirm, Clear, Call, Cordon, Control (5 C’s) 

4. B When considering the effects of terrain, which is not one of the 

standard considerations as described by the acronym OCOKA? 

a. Observation and fields of fire 

b. Orientation 

c. Cover and concealment 

d. Avenues of approach 

5. B Which of the following is a way that short and long security halts 

are the same? 

e. Amount of dismounted support employed 

f. Vehicle crew served weapons are manned and oriented on 
designated sector of fire 

g. Amount of cover employed by dismounts 

h. Vulnerability of main body 

6. D Where in the convoy should the convoy commander place himself? 

e. Lead vehicle 

f. Trail vehicle 

g. With the security element commander 

h. Main body 

7. C Who is an appropriate person to conduct casualty reporting? 

e. Combat life saver 

f. Corpsman 

g. Assistant convoy commander 

h. Security element commander 

8. D Which of the following correctly represents the “T’s” in METT-T? 

a. Training, Troops, Time 

b. Terrain, Training, Time 

c. Terrain, Trucks, Time 

d. Terrain, Troops, Time 


lA Procedure 

Unblocked 
ambush—does 
student 

understand basic 
execution? 


Mission planning 

Does student 
understand the 
purpose of 

convoy? 


lA Procedure 

lED—does the 

student 

understand the 5 
C’s? 


Mission planning 

METT-T— 
Understanding of 
terrain analysis 
steps 


lA Procedure 

Halts—crew 
served weapons 
must always be 
employed in 
security as priority 


Responsibilities 

Does the student 
understand CC 
enough not to be 
in front lOT 
maintain control? 


Responsibilities / 
Division of labor 

Does student 
understand to 
delegate reporting 
so CC can fight? 


Mission planning 

METT-T— 
Understanding of 
basic mission 
planning steps 
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9. D When a convoy vehicle first becomes disabled, the security element 

commander should consider the following as his primary responsibility: 

e. Reporting the disabled vehicle to higher headquarters 

f. Punching forward to determine whether enemy are waiting 
to ambush 

g. Ensuring the convoy stops in a herringbone in case of air 
attack 

h. Providing security for the convoy main body 


lA Procedure 

Vehicle rec—does 
the student 

understand to 
maintain convoy 
security first? 


10. D In the 5 paragraph operational order, in which paragraph would 

you find the frequency of the higher command radio net? 

a. Paragraph 2 

b. Paragraph 3 

c. Paragraph 4 

d. Paragraph 5 

11. D Who specifies the speed of the convoy? 

e. Lead vehicle commander 

f. Security element commander 

g. Driver of the lead vehicle 

h. Convoy Commander 


Mission planning 

OPORD— 
Understanding of 
basic OPORD 
structure 


Responsibilities 

Does the student 
understand that 
the logistics in 
main body 

determines spd? 


12. C Which of the following is not one of the troop leading steps? 

a. Supervise 

b. Complete the plan 

c. Execute the operation 

d. Begin planning 

13. A Which action should the convoy commander avoid when 

reacting to a sniper? 

e. Stop 

f. Speed up 

g. Return fire 

h. Use smoke 

14. B In order to maintain flexibility in the face of a potential threat, a 

convoy commander should: 

e. Focus on speed and momentum throughout the convoy 

f. Plan an alternate route 

g. Place himself in the lead of the convoy 

h. Place himself in the trail of the convoy 

15. B Which is one of the three elements of a convoy task organization? 

e. Trail 

f. Security 

g. Main body 

h. Head 


Mission planning 

BAMCIS— 
Understanding of 
basic troop 

ieading steps 


lA Procedure 

Sniper—does the 
student 

understand basic 
execution? 


Mission planning 

Does the student 
understand how 
mission pianning 
affects future 
execution? 


Organizational 
structure vs task 
organization 

Does the student 
understand task 
organization? 


1 6. D What part of the operational order would you use to ensure your 
subordinates understood the final end state of the operation? 

a. Concept of operations 

b. Coordinating instructions 

c. Command and signal 

d. Commander’s intent 


Mission planning 

OPORD— 
Understanding of 
basic OPORD 
structure 
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17. B Who maintains communication with higher and adjacent 

authorities for the convoy? 

e. Vehicle commander 

f. Convoy commander 

g. Assistant convoy commander 

h. Security force commander 

18. B Why would you use METT-T? 

a. Describe the effects of terrain on the operation 

b. Analyze your mission in order to plan the operation 

c. Develop the time line for the operation 

d. Coordinate indirect and direct fire plans for the concept 
of operations 

19. B It would be inappropriate to speed up for which immediate 

action drill? 

e. Reaction to sniper 

f. Reaction to unexploded I ED 

g. Reaction to unblocked ambush 

h. All of the above 

20. B When crossing a danger area, which element provides over 

watch while the main body crosses? 

e. Obstacle Clearing Detachment 

f. Trail 

g. Main body’s security elements 

h. Combat Assault Element 


Responsibilities / 
Division of labor 

Does student 
understand to 
delegate reporting 
and CC fight? 


Mission planning 

METT-T— 
Understanding the 
purpose of METT- 
T as an aid to 
mission planning 


lA Procedure 

Ambush—does 
the student 

understand basic 
execution? 


lA Procedure 

Danger area— 
does student 

understand basic 
execution 


21. B Which part of the 5 paragraph operational order paints a word 
picture of how the operation should unfold? 

a. Terrain and weather 

b. Concept of operations 

c. Friendly situation 

d. Mission 


Mission planning 

OPORD— 
Understanding of 
basic OPORD 
structure 


22. D The first step in reacting to an lED is: 

a. Cordon 

b. Report the lED to higher headquarters 

c. Call EOD 

d. Confirm 


lA Procedure 

lED—does the 

student 

understand the 5 
C’s? 


23. A 


In the 5 paragraph operational order, in which paragraph would you 
find the enemy’s most likely course of action? 

a. Paragraph 1 

b. Paragraph 2 

c. Paragraph 3 

d. Paragraph 4 


Mission planning 

OPORD— 
Understanding of 
basic OPORD 
structure 


24. B 


Which of the following should not be used to determine convoy speed? 

e. Terrain 

f. Need for crew rest 

g. Weather conditions 

h. Likelihood of enemy contact 


Movement 

Does the student 
understand basic 
movement 
prinoiples? 
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